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CAMERA SYSTEM FOR HIGH-SPEED IMAGE PROCESSING 



TECHNICAL FIELD 

5 The present invention relates to a high-speed camera 

system having an image processing function. 

BACKGROUND ART 

D High-speed image processing is required to operate 

So robots at a high speed in a factory automation (FA) system 

H 5 or the like. For example, when a robot is configured with a 

§ 

fu feedback loop between its vision sensor and xts actuator, 

a 

O the actuator can be controlled in units of milliseconds. 

W 

nJ Accordingly, the image processing speed that matches this 

S3 

£3.5 speed is required. However, because the image processing 

ru 

speed in modern vision systems is limited by the video frame 
rate, the robot can only operate at low speeds that match 
this image processing speed. It is impossible to take full 
advantage of the robot's performance. Inspection systems 

20 with using vision sensors is limited in their accuracy and 
speed because of the restrictions of the video frame rate. 

There are some high-speed CCD cameras that can take 
images within about one (1) millisecond. In these devices, 
the images are temporarily stored in memory and later read 

2 5 and processed. Therefore, the devices can be applied to 
such applications as image analysis. However, the devices 
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have almost no real-time capability, and are not suitable 
for controlling robots, on-line inspections, or other 
applications . 

In order to overcome these problems, institutes such 
5 as the Massachusetts Institute of Technology, the California 
Institute of Technology, and Mitsubishi Electric Corporation 
have been researching a vision chip that combines the image 

M, reading portion and the processing portion into one unit 

Q 

□ ("An Object Position and Orientation IC with Embedded 

CO 

5.0. Imager," David L. Standley, Solid State Circuits, Vol. 26, 

M . 

go No. 12, Dec. 1991, pp. 1853-1859, IEEE); "Computing Motion 

i ig 

3 Using Analog and Binary Resistive Networks," James 

O 

hi Hutchinson, et al . , Computer, Vol. 21, March 1988, pp. 52-64, 

ru 

%J IEEE) ; and "Artificial Retinas — fast versatile image 

r§5 processors," Kazuo Kyuma, et al . , Nature, Vol. 372, November 
10, 1994) . However, these chips employ a fixed analog 
circuit that is easy to integrate. Accordingly, these 
circuits have various shortcomings. That is, the circuits 
reguire subseguent-processing of output signals. The 
20 circuits suffer from the problems specific to analog signals. 
For example, the circuits are reguired to cope with noise 
and to attain a desired signal-to-noise ratio. The type of 
image processing they can perform is limited to special 
applications. The circuits have a lack of universality 
2 5 because the contents of the image processing are limited to 
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particular applications. 

To overcome these problems, a technology for a vision 
sensor capable of performing general purpose high-speed 
image processing is disclosed in Japanese examined patent 
application publication no. HEI-7-62866 (hereinafter 
referred to as conventional technology no. 1). For further 
improving integration, a technology for optimizing transfer 
lines in each row has been proposed in Japanese unexamined 
patent application publication HEI-10-145680 (hereinafter 
referred to as conventional technology no. 2). These types 
of vision sensor increase processing speed by providing a 
plurality of processing elements in one to one 
correspondence with a plurality of photodetector elements to 
achieve complete parallel processing. A technology for 
constructing a high-resolution sensor through partial 
parallel processing by providing one processing element for 
each array of photodetector elements is disclosed in 
International Publication W095/24793 (hereinafter referred 
to as conventional technology no. 3). When used in 
inspection equipment on a production line, this type of 
vision sensor can greatly reduce the length of testing time. 

In the actual use of these sensors, however, it is 
frequently necessary for a human being to visually confirm 
images while the images are being processed. When examining 
products on a belt conveyor, for example, a human worker is 



often entrusted with the final check. Hence, the image must 
be outputted to a display device that the worker can view. 
When setting and adjusting the equipment, in order to grasp 
how the inspection is achieved, visual confirmation is also 
5 necessary for checking lighting conditions on the image pick 
up position, lens focus, stains on the lenses, and other 
conditions. An efficient method of establishing processing 
q algorithms for inspection methods is to set threshold levels 

gj and matching points while monitoring the processes by 

ylO viewing actual images. It is possible to efficiently 

In 

m estimate the processing algorithms. 

p However, devices conforming to conventional 

111 

Sf= technologies nos . 1 through 3 do not have an efficient 

M 

function for outputting acquired images to external devices . 

Ill 

= I5 These devices can only output results of calculations, in 
the form of characteristic quantities (such as the object's 
center of gravity and inspection results) or in the form of 
control signals (for controlling the motor and the like) . 
Even if those devices are modified to output the acquired 

2 0 images, the frame rate of the images outputted from the 
high-speed camera is extremely fast and cannot be supported 
by a normal television monitor. Accordingly, a special 
display device is required to output the images, thereby 
increasing the cost of the system. Further, even if the 

25 images outputted from the high-speed camera can be displayed 
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as they are, necessary details of the image may not be 
perceived by the human eye, which has a limited response 
rate (about 30 milliseconds) . 

A device for monitoring fast moving objects at a high- 
resolution has been disclosed in Japanese unexamined patent 
application publication HEI-10-21407 (hereinafter referred 
to as conventional technology no. 4). This monitor operates 
by overlaying image data from the high-speed image- 
processing device on image data from a conventional image- 
processing device. However, the frame rate of images 
outputted from the monitor of conventional technology no. 4 
is limited to nearly the same video rate as the conventional 
rate. Therefore, it is difficult to extract the reguired 
information when observing fast moving objects. 

Sometimes it is important to capture images at a 
particular timing, such as the instant of a press or the 
penetration of a drill performed by automated machine tools, 
the instant a part is inserted, the moment a golf club or a 
baseball bat meets a ball, and the like. It is difficult to 
display such images at the appropriate timing and in a 
manner that can be perceived by the human eye. 



DISCLOSURE OF THE INVENTION 
In • view of the foregoing, it is an object of the 
present invention to solve the above-described problems, and 



to provide a camera system for high-speed image processing 
which is capable of performing image processing at a high 
rate of speed and which is capable of outputting images that 
are picked up at a particularly important timing. 

In order to overcome the above-described problem and 
other problems, the present invention provides a camera for 
high-speed image processing, comprising: a photodetector 
array having a plurality of photodetectors , which are 
arranged two-dimensionally in a plurality of rows and in a 
plurality of columns and which are divided into a plurality 
of blocks, the photodetector array repeatedly receiving 
optical images and generating output signals for the optical 
images in a plurality of consecutive frames at a 
predetermined frame rate; an analog-to-digital converter 
array including a plurality of analog-to-digital converters 
in one-on-one correspondence with the plurality of blocks in 
the photodetector array, each analog- to-digital converter 
performing analog-to-digital conversion of the output 
signals which are read seguentially from the photodetectors 
in the corresponding block; an image-processing unit 
performing predetermined parallel processes on digital 
signals, which are transferred from the analog-to-digital 
converter array and which correspond to the signals 
outputted from the photodetectors , thereby generating a 
processed result signal indicating the results of the 



processes; a selector selecting at least one frame based on 
the processed results obtained by the image-processing unit; 
a signal converter converting, into an image signal of a 
desired frame rate, at least one of the processed result 
signal and the output signal from the analog-to-digital 
converter array, the signal converter outputting the image 
signal; and a signal conversion controller controlling the 
signal converter to perform the image signal conversion 
operation for the at least one frame selected by the 
selector . 

In the camera system for high-speed image processing 
according to the present invention, the image-processing 
unit performs parallel processes, thereby attaining high- 
speed processing in real-time. Moreover, the camera system 
includes the signal converter. The signal converter 
converts, into image signals of a desired frame rate, at 
least one of the processed result signals from the image- 
processing unit and the signal from the analog-to-digital 
converter array. The signal converter outputs the image 
signals. Thus, the camera system extracts a desired image 
based on the processing results by the image-processing unit, 
and converts, into image signals of the desired frame rate, 
at least one of the processed result signals for the 
extracted image and the output signal from the analog-to- 
digital converter array for the extracted image. 



Accordingly, by outputting at least one of the above signals 
at an appropriate frame rate, it is possible to output an 
image for a special timing at an appropriate frame rate. 
For example, by appropriately displaying at least one of the 
above signals for the extracted image on a monitor, an image 
of an important timing can be monitored. Also, by 
appropriately outputting at least one of the above signals 
for the extracted image to an image-processing device in a 
later stage, it is possible to supply an image of an 
important timing for further image processing. Accordingly, 
the present invention can perform high-speed image 
processing through parallel processes and output the 
processed result at a desired timing. 

It is preferable that the desired frame rate is lower 
than the predetermined frame rate. In this case, the signal 
converter downconverts at least one of the processed result 
signals from the image-processing unit and the signal 
outputted from the analog-to-digital converter array, into 
an image signal of a desired low frame rate supported by a 
monitor, which is used "to display images, or supported by an 
image-processing unit for performing further image processes, 
and outputs the image signal. It is possible to display the 
image, which has been processed at a high speed, on a 
monitor at an appropriate frame rate and for a desired 
timing required for performing visual inspections or the 



like. It is therefore possible to display only the images 
that need attention and therefore that have been selected 
from all the images. Further, by outputting images, which 
have been processed at a high speed, at a desired frame rate 
for a timing required for image processing or the like, it 
is possible to extract and output only the images that need 
attention . 

It is preferable that the signal converter converts 
either one of the output signal from the analog-to-digital 
converter array and the processed result signal, into the 
image signal of the desired frame rate, and outputs the 
image signal. Hence, it is possible to extract desired 
images from among the raw images obtained by the 
photodetector array, per se. and the images obtained through 
processing by the image-processing unit, and to output the 
extracted desired images. Accordingly, it is possible to 
extract and display only images that need attention, for 
example . 

The signal converter may combine the output signal 
from the analog-to-digital converter array and the processed 
result signal, convert the combined signal into the image 
signal of the desired frame rate, and output the image 
signal. For example, with this construction, it is possible 
to extract desired images from the raw images, obtained by 
the photodetector array, and to display those images along 



with processing results obtained by the processing of the 
extracted desired images. For example, the raw image and 
the processed result image can be displayed next to each 
other, top and bottom or left and right. Accordingly, the 
present invention achieves high-speed image processing using 
parallel processes, and displays the processed results 
superimposed over the original image on a monitor at a 
desired timing. For example, it is possible to extract only 
images that need attention and emphasize those images in the 
monitor display. 

It is preferable that the signal converter has a 
buffer memory at a signal input side thereof, the buffer 
memory storing at least one of the output signal from the 
analog-to-digital converter array for at least several 
frames and the processed result signal for the at least 
several frames. With this construction, it is possible to 
extract a plurality of images, such as consecutive images, 
at a special timing, and to output the extracted images 
thereafter. By providing the signal converter with the data 
buffer capable of storing image data for a plurality of 
frames, it is possible to display the video images around a 
desired timing while slowing down the display speed. 

It is preferable that the camera for high-speed image 
processing according to the present invention further 
comprises a data buffer storing predetermined process data, 
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the image-processing unit performing a predetermined 
parallel process, using the predetermined process data, onto 
the digital signals that are transferred from the analog-to- 
digital converter array and that correspond to the signals 
outputted from the photodetectors . By using the 

predetermined process data stored in the data buffer, it is 
possible to perform processes, particularly comparison 
processes and the like, at a high speed. 

It is preferable that the plurality of analog-to- 
digital converters in the analog-to-digital converter array 
are provided in one-on-one correspondence with the plurality 
of rows or the plurality of columns of photodetectors in the 
photodetector array. This configuration facilitates data 
transfer to processing elements, and circuit design. 

It is preferable that the image-processing unit 
includes a plurality of processors in one-to-one 
correspondence with the plurality of photodetectors, the 
plurality of processors performing parallel processes on the 
digital signals that are transferred from the analog-to- 
digital converter array and that correspond to the signals 
outputted from the plurality of photodetectors. In this way, 
processes are performed only once on the plurality of pixels 
simultaneously, to thereby achieve image processing at a 
higher rate of speed. 

It is preferable that the image-processing unit 



includes at least one parallel processing circuit, each of 
the at least one parallel processing circuit performing a 
corresponding parallel process on the digital signals that 
are transferred from the analog- to-digital converter array 
and that correspond to the signals outputted from the 
photodetectors , thereby outputting a processed result signal 
indicative of the processed result, the selector selecting 
at least one frame based on at least one process result from 
the at least one parallel processing circuit, the signal 
converter converting at least one of the output signal from 
the analog-to-digital converter array and the processed 
result signal obtained by the at least one parallel 
processing circuit, into the image signal of the desired 
frame rate, and outputting the image signal. 

In this case, it is preferable that each of the at 
least one parallel processing circuit is provided with 
several processing elements in one-to-one correspondence 
with several blocks that make up at least a portion of all 
the plurality of blocks in the photodetector array, each of 
the several processing elements performing a predetermined 
parallel process on the digital signals that are transferred 
from the corresponding analog-to-digital converter and that 
are equivalent to the signals outputted from the 
photodetectors existing in the corresponding block. 

When the plurality of analog- to-digital converters in 
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the analog-to-digital converter array are provided in one- 
to-one correspondence with the plurality of rows of 
photodetectors in the photodetector array, it is preferable 
that each of the at least one parallel processing circuits 
5 includes several processing elements which are provided in 
one-to-one correspondence with several rows that make up at 
least a portion of all the plurality of rows in the 

p photodetector array, each of the several processing elements 

O 

g performing a predetermined parallel process on digital 

JjTO signals that are transferred from the analog-to-digital 

ill 

si converter array and that are eguivalent to the signals 

s 

p outputted from the photodetectors in the corresponding row. 

W 

ru 
M 

q BRIEF DESCRIPTION OF THE DRAWINGS 

ftj 

15 Fig. 1 shows the construction of a camera system for 

high-speed image processing according to a first embodiment 
of the present invention; 

Fig. 2 shows the general configuration of the camera 
system for high-speed image processing according to the 
20 first embodiment; 

Fig. 3 is a circuit diagram for a photodetector array 
and analog-to-digital converter array in the camera system 
according to the first embodiment; 

Fig. 4 is a detailed circuit diagram for an 
25 integrating circuit in the camera system according to the 
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first embodiment; 

Fig. 5 is a block diagram showing a detailed 
construction of a signal converter in the camera system 
according to the first embodiment ; 

Fig. 6 is a block diagram of a processing circuit in 
the image-processing unit of the camera system according to 
the first embodiment ; 

Fig. 7 is a block diagram of a center-of-gravity 
calculating circuit in the image-processing unit of the 
camera system according to the first embodiment; 

Fig. 8 is a block diagram of an edge extraction 
circuit in the image-processing unit of the camera system 
according to the first embodiment; 

Fig. 9 is an explanatory diagram showing how the 
signal converter is connected to the analog-to-digital 
converter array and each processing circuit in the image- 
processing unit in the camera system according to the first 
embodiment; 

Fig. 10 is an explanatory diagram showing the concept 
of image processing in the camera system according to the 
first embodiment; 

Figs. 11 (a) -11(c) are timing charts of image data 
signals showing the overlap of raw images and processed 
images, wherein Fig. 11(a) shows the timing of raw image 
data, Fig. 11(b) shows the timing in which processed signals 



are combined with the raw image data, and Fig. 11(c) shows 
another example of the timing in which processed signals are 
combined with the raw image data; 

Fig. 12 is a flowchart showing the process of control 
in the camera system according to the first embodiment ; 

Fig. 13(a) is a timing chart for image processing and 
image display with the camera system of the first 
embodiment ; 

m Fig. 13 (b) is a timing chart for image processing and 

|£0 image display according to the prior art; 

s 

pj Figs. 14 (a) -14(c) are explanatory diagrams showing how 

□ an image is extracted from consecutive images, wherein Fig. 

liJ 

m 14 (a) is an image prior to a desired extraction timing. Fig. 

Q 14 (b) is an image at the desired extraction timing, and Fig. 

T5 14 (c) is an image after the desired extraction timing; 

Fig. 15 is a block diagram showing a modification of 
the processing circuits in the image-processing unit of the 
camera system according to the first embodiment; 

Fig. 16 is an explanatory diagram showing how the 
20 signal converter is connected to each processing circuit in 
Fig. 15; 

Fig. 17 shows the configuration of a camera system for 
high-speed image processing according to a second embodiment 
of the present invention; 
25 Fig. 18 is a block diagram of processing circuits in 
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the image-processing unit of the camera system according to 
the second embodiment; 

Fig. 19 is an explanatory diagram showing how the 
signal converter is connected to each processing circuit in 
the second embodiment ; 

Fig. 20 is an explanatory diagram showing the concept 
of image processing in the camera system according to the 
second embodiment; 

Fig. 21 shows the configuration of a camera system for 
high-speed image processing according to a third embodiment 
of the present invention; 

Fig. 22 is a block diagram of the processing -circuits 
in the image-processing unit of the camera system according 
to the third embodiment ; 

Fig. 23 is a block diagram of the pattern matching 
circuit in the processing circuits in Fig. 22; 

Fig. 2 4 is an 'explanatory diagram showing how the 
signal converter is connected to the analog-to-digital 
converter array and each processing circuit in the third 
embodiment ; 

Fig. 25 is a flowchart showing the process of control 
in the camera system according to the third embodiment; 

Fig. 26 is an explanatory diagram showing the concept 
of image processing in the camera system according to the 
third embodiment; 



Fig. 27(a) shows the configuration of a camera system 
for high-speed image processing according to a fourth 
embodiment of the present invention; 

Fig. 27(b) is a flowchart showing the process of 
5 control in the camera system according to the fourth 
embodiment ; 

Fig. 2 8 is a block diagram showing a modification of 
~ the camera system according to the present invention ; 

5; Fig. 2 9 is a block diagram showing another 

50 modifidation of the camera system according to the present 

51 invention; 

~ Fig. 3 0 is a block diagram showing another 

Sr} modifidation of the camera system according to the present 

j~ invention; and 

m 

! I5 Fig. 31 is a block diagram showing another 

modifidation of the camera system according to the present 
invention. 

BEST MODE FOR CARRYING OUT THE INVENTION 
2 0 A camera system for high-speed image processing 

according to preferred embodiments of the present invention 
will be described while referring to Figs. 1 through 27. 

Like parts and components are designated by the same 
reference numerals to avoid duplicating description. 
25 First, a camera system for high-speed image processing 
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according to a first embodiment of the present invention 
will be described with reference to Figs. 1 through 16. 

Fig. 1 is a block diagram showing a camera system 1 
for high-speed image processing according to the first 
embodiment of the present invention. 

The camera system 1 of the present embodiment 
includes: a photodetector array 11; an analog-to-digital 
(A/D) converter array 13; an image-processing unit 14; a 
control circuit 15; an instruction/command bus 16; a signal 
converter 17; a signal conversion controller 19; and a 
monitor 18. The photodetector array 11 includes Nl x N2 
photodetector elements 12 0 which are arranged two- 
dimensionally . The analog-to-digital converter array 13 
includes N2 analog-to-digital converters 210. Each analog- 
to-digital converter 210 corresponds to one row of the 
photodetector array 11, and serves to convert the electric 
charge outputted from photodetectors , that belong to the 
corresponding row, into voltage signals and to perform 
analog-to-digital conversion on the voltage signals. The 
image-processing unit 14 is provided with a processing 
element array. The processing element array includes a 
plurality of (three, in the present example) processing 
circuits 400. The control circuit 15 is for controlling all 
the circuits by sending commands and the like thereto. The 
instruction/command bus 16 is for transmitting signals from 
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the control circuit 15 to all the circuits. The signal 
converter 17 is for selectively accumulating data signals 
transmitted from the image-processing unit 14 and converting 
the signals into image display signals. The signal 
conversion controller 19 is for controlling the signal 
converter 17 to perform the signal accumulation and the 
signal conversion. The monitor 18 is for displaying images 
by using the image display signals obtained by conversion at 
the signal converter 17. The monitor 18 is a television 
monitor, for example, that provides image displays for 
visual inspection. It is also possible to connect a 
computer, in place of the monitor 18, to the signal 
converter 17. In this case, signals selectively converted 
by the signal converter 17 are outputted directly to the 
computer as output signals. The computer can perform image 
processing using conventional image processing techniques on 
the results of calculations executed by the image-processing 
unit 14. 

Fig. 2 shows the structure of an example of the camera 
system 1 according to the present embodiment. In this 
figure, semiconductor chips, such as the photodetector array 
11, the analog-to-digital converter array 13, the image- 
processing unit 14, the signal converter 17, the signal 
conversion controller 19, and the control circuit 15 are 
bonded directly on a single substrate. The camera system 1 



can therefore be manufactured with a high level of 
integration. In addition, it is possible to employ a 
processing method suitable for the characteristics of each 
device, thereby enabling a stable production. In the future, 
according to the advance of the processing technology, it 
will become possible to integrate all the functions on one 
chip . 

Next, the internal structure of each circuit will be 
described . 

Fig. 3 shows a detailed construction of an image input 
portion. The image input portion includes: a light 
receiving portion 100, a signal processing portion 200, and 
a timing control portion 300. The light receiving portion 
100 is for detecting light. (The light receiving portion 
100 corresponds to the photodetector array 11 in Fig. 1.) 
The signal processing portion 200 is for processing output 
signals from the light receiving portion 100. (The signal 
processing portion 200 corresponds to the analog-to-digital 
converter array 13 in Fig. 1.) The timing control portion 
300 is for transmitting instruction signals, indicative of 
operation timings, to both of the light receiving portion 
100 and the signal processing portion 200. (The timing 
control portion 300 corresponds to a part of the control 
circuit 15 in Fig. 1.) 

First, with reference to Fig. 3, the structure of the 
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light receiving portion 100, which corresponds to the 
photodetector array 11 in Fig. 1, will be described. 

One photodetector 120 is configured from a 
photoelectric conversion element 130 and a switch element 
5 140. The photoelectric conversion element 130 is for 
generating an electric charge according to the intensity of 
received light. The switch element 140 is connected to a 

Q signal output terminal of the photoelectric conversion 

~- 

m element 130. The switch element 140 outputs the electric 

§40 charge, accumulated in the photoelectric conversion element 

ril 130, in response to a horizontal scan signal Vi (i = 1 ~ Nl) . 

O One horizontal photodetector row 110 is constructed from Nl 

y 

|1| photodetectors 12 0 which are arranged in a first direction 

p (which will be referred to as "horizontal direction") with 

ru 

15 their switch elements 140 being connected with one another. 

The light receiving portion 100 is constructed from N2 
horizontal photodetector rows 110 which are arranged 
vertically, that is, perpendicularly to the horizontal 
direction . 

20 Next, the structure of the signal processing unit 200, 

which corresponds to the analog-to-digital converter array 
13, will be described with reference to the same figure 3. 

The signal processing unit 200 includes N2 analog-to- 
digital converters 210-; (j=l~N2). Each analog-to-digital 

25 converter 210j is for individually extracting electric 
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charges transmitted from the corresponding horizontal 
photodetector row 110j (j=l ~ N2) , for processing the 
electric charges, and for outputting digital signals 
corresponding to the magnitudes of the electric charges. 
Each analog-to-digital converter 210j includes: an 
integrating circuit 220j having a charge amp 221j; a 
comparator circuit 230j ; and a capacity control mechanism 
240j . 

The integrating circuit 220j includes: the charge amp 
221j; a variable capacity unit 222j ; and a switch element 
223j . The charge amp 221j is for amplifying the charge of an 
input signal received from the horizontal photodetector row 
110j. The variable capacity unit 222j is connected, on one 
end, to the input terminal of the charge amp 221j, and is 
connected, on the other end, to the output terminal of the 
same. The switch element 223j is connected, on one end, to 
the input terminal of the charge amp 22 lj, and is connected, 
on the other end, to the output terminal of the same. The 
switch element 223j is turned ON or OFF in response to a 
reset signal R, thereby toggling the operation of the 
integrating circuit 22 Oj between an integrating state and a 
not-integrating state. 

Fig. 4 shows a more detailed construction of the 
integrating circuit 220. This drawing shows one example of 
the integrating circuit that has an analog-to-digital 
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converting function with a 4-bit (16 gradation) resolution. 
The integrating circuit 220 will be described below for this 
circuit structure. 

The variable capacity unit 222 includes: capacity 
elements CI - C4; switch elements SWll - SW14; and other 
switch elements SW21 - SW2 4. Each of the capacity elements 
CI - C4 is connected, on one end, to the input terminal of 
the charge amp 221 that receives signals outputted from the 
corresponding horizontal photodetector row. Each of the 
switch elements SWll - SW14 is connected between the other 
end of a corresponding capacity element CI - C4 and the 
output terminal of the charge amp 221. The switch elements 
SWll — SW14 are for opening and closing in response to 
capacity instructions signals Cn - C i4 , respectively. Each 
of the switch elements SW21 - SW2 4 has one terminal, which 
is connected between the corresponding capacity element CI - 
C4 and the corresponding switch element SWll - SW14, and 
another terminal, which is connected to a ground level GND . 
The switch elements SW21 - SW2 4 are for opening and closing 
in response to capacity instructions signals C 2 i - C 2 4, 
respectively. The capacities Ci - C 4 of the capacity 
elements CI - C4 have the following relationships: 

Ci = 2C 2 = 4C 3 = 8C 4 

Co = Ci + C 2 + C 3 + C 4 

Wherein C 0 is the maximum electrical capacity required 



23 



by the integrating circuit 220. It is assumed that the 
saturation charge of the light receiving element 13 0 (see 
Fig. 3) is Q 0 and that a reference voltage is Vre F . In this 
case, the following relationship is satisfied: 
Co = Qo/Vref 

The remainder circuits in the analog-to-digital 
converter 210j other than the integrating circuit 220j will 
be described below while once again referring to Fig. 3. 
The comparator circuit 230j is for comparing the value of an 
integral signal V s , which is outputted from the integrating 
circuit 220j, with the reference voltage Vref, and for 
outputting a comparison result signal V c . Based on the 
value of the comparison result signal V c , the capacity- 
control mechanism 240j outputs a capacity instruction signal 
C to notify the variable capacity unit 222-j in the 
integrating circuit 220j. The capacity control mechanism 
240j also outputs, to a data transfer line 250, a digital 
signal Dl that corresponds to the capacity instruction 
signal C. In this way, photoelectric conversion output 
obtained at each photodetector 120 (i,j) is converted into a 
digital signal D(x,y) (where (x,y) = (i,j)), which is a raw 
image signal indicative of image information. 

The output terminal of the capacity control mechanism 
240j in each analog-to-digital converter 210j is connected 
to a corresponding data transfer line 250j . Accordingly, N2 
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data transfer lines 250j are connected to the output 
terminal of the signal processing unit 200 (analog-to- 
digital converter array 13) . As shown in Fig. 1, the data 
transfer lines 250j are connected to the image processing 
5 unit 14. The data transfer lines 250j are connected also to 
the raw image data buffer 460. N2 data transfer lines 500j 
are connected to the output terminal of the raw image data 
O buffer 460. The raw image data buffer 460 is for storing 

fn the digital signals Dl , which are successively inputted from 

tiO each data transfer line 250j, for a predetermined period of 
HJ time, and thereafter transferring those digital signals Dl 

3 

O to the signal converter 17 via a corresponding data transfer 

w 

llj line 500-j. 

SI 

p It is noted that the analog-to-digital converter array 

ru 

15 13 has been described above for the case where the analog- 
to-digital converter array 13 has a 4-bit (16 gradation) 
resolution. However, the analog-to-digital converter array 
13 can be configured to have a 6-bit, 8-bit, or other bit 
resolution . 

20 Next will be described the structure of the timing 

control portion 300 shown in Fig. 3. The timing control 
portion 300 includes: a basic timing portion 310, a 
horizontal shift register 320, and a control signal portion 
340. The basic timing portion 310 is for generating a basic 

25 timing to attain clock control of the entire circuit. The 
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horizontal shift register 320 is for generating a horizontal 
scan signal V± according to a horizontal scan instruction 
inputted from the basic timing portion 310. The control 
signal portion 340 is for generating a reset instruction 
5 signal R. 

In the image-processing unit 14, the plurality of 
processing circuits 400 are connected in cascade or in 
Q parallel with one another. The analog-to-digital converter 

ffi array 13 outputs, for each row j, digital signals in order 

yy 

§40 from the most significant bit (MSB) . The image-processing 

a 

fy unit 14 conducts a predetermined image process on the 

s 

O digital signals thus transferred from the analog-to-digital 

w 

rU converter array 13 . 

M 

0 The image process performed by each processing circuit 

•== 

1 y 

15 400 can be implemented in either hardware or software. The 
present embodiment employs processing circuits that perform 
calculations in hardware. 

More specifically, a binarization circuit 401, a 
center-of-gravity calculating circuit 402, and an edge 

20 extraction circuit 403 are provided as the processing 
circuits 400. Each of the circuits 401, 402, and 403 has a 
special construction suited for its particular process. 
Here, a field programmable gate array (FPGA) or the like, 
which facilitates the development and mounting of hardware 

25 for each image process, can be used in order to efficiently 
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perform setting processes for each objective in hardware. 
Further, it is now possible to design circuits in software 
using HDL {Hardware Description Language) , making it easier 
to apply conventional knowledge. 

As shown in Fig. 5, the image-processing unit 14 and 
the raw image data buffer 4 60 are connected to the signal 
converter 17 via N2 switches 20. The switches 20 enable the 
results of processes by the processing circuits 400 in the 
image-processing unit 14 and the raw image data to be 
combined as they are supplied to the signal converter 17. 

Below, a more detailed description is provided for the 
binarization circuit 401, the center-of-gravity calculating 
circuit 402, and the edge extraction circuit 403 in the 
image-processing unit 14. In the present embodiment, the 
center-of-gravity calculating circuit 402 and the edge 
extraction circuit 403 are connected in parallel with each 
other in the later stage of the binarization circuit 401. 

First the binarization circuit 401 will be described 
with reference to Fig. 6. In the binarization circuit 401, 
N2 bit comparators 405j are provided in one-to-one 
correspondence with the N2 rows in the photodetector array 
11. One input terminal of each bit comparator 405j is 
connected to the corresponding data transfer line 250j from 
the analog-to-digital converter array 13. The other input 
terminal of the bit comparator 405j is connected to the 



control circuit 15 via the instruction/command bus 16. The 
other input terminal is inputted with a data signal for the 
object of comparison. The output terminal of each bit 
comparator 405j is connected to a data transfer line 440j . 
Each bit comparator 405j outputs its comparison results to 
the corresponding data transfer line 440j. In this way, 
each data transfer line 440j is connected, via the analog- 
to-digital converter array 13, to one pixel line of the 
corresponding row j in the photodetector array 11. 

As shown in Fig. 6, the center-of-gravity calculating 
circuit 402, used to perform center-of-gravity calculations, 
is connected to the N2 data transfer lines 440j from the 
binarization circuit 401. The center-of-gravity calculating 
circuit 402 serves to calculate the center of gravity, a 
basic process in image processing. The center-of-gravity 
calculating circuit 402 includes a center-of-gravity 
detecting circuit 409 and a center-of-gravity image data 
generating circuit 425. 

The construction of the center-of-gravity detecting 
circuit 409 will be described in detail below with reference 
to Fig. 7. 

The center-of-gravity detecting circuit 409 has N2 
adders 410j that correspond to the N2 rows of the 
photodetector array 11, that is, the N2 analog- to-digital 
converters 210j. In other words, the N2 data transfer lines 
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440-j extending from the binarization circuit 401 are 
connected to the N2 adders 410j, respectively. Each adder 
410j is connected to one input terminal of a corresponding 
multiplier 411j . A row position data supply circuit 412 is 
5 connected to the other input terminal of each multiplier 
411j. The output terminals of all the N2 multipliers 411j 
are connected to N2 input terminals of a single adder 413. 

U 

p The output terminal of the adder 413 is connected to the 

fg dividend input terminal of a divider 414. 

MLO The N2 data transfer lines 440j from the binarization 

ry circuit 401 are also connected to N2 input terminals of 

p another single adder 415. The output terminal of the adder 

415 is connected both to: a single input terminal of another 
3 adder 418 and an input terminal of a multiplier 416. A 

y 

15 column position data supply circuit 417 is connected to the 
other input terminal of the multiplier 416. The output 
terminal of the multiplier 416 is connected to an input 
terminal of an adder 419. The output terminal of the adder 
419 is connected to the dividend input terminal of a divider 

20 420. The output terminal of the adder 418 is connected to a 
divisor terminal of each of the dividers 414 and 420. The 
quotient (divided result) determined by the dividers 414 and 
42 0 indicate the y coordinate and x coordinate for the 
center-of-gravity position (Px, Py) in a single frame image 

25 denoted by the Nl x N2 input signals D(x, y) . Data transfer 
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lines 510y, and 510x are connected to the output terminals 
of the dividers 414 and 420, respectively. The dividers 414 
and 420 output the center-of -gravity position data Py and Px 
via the corresponding data transfer lines 510y and 510x. 

As shown in Fig. 6, the data transfer lines 510y and 
510x from the dividers 414 and 42 0 are connected to the 
control circuit 15 via the instruction/ command bus 16 in 
order to output the center-of-gravity position data (Px, Py) 
-to the control circuit 15. The data transfer lines 510y and 
510x are also connected to the center-of-gravity image data 
generating circuit 425. 

The center-of-gravity image data generating circuit 
425 functions to generate center-of-gravity image data P(x, 
y) indicating a point corresponding to the position of the 
center of gravity based on the quotient from the dividers 
414 and 420, that, is, the center-of-gravity position (Px, 
Py) of the input image data. N2 data transfer lines 52 0j 
are connected to the output terminal of the center-of- 
gravity image data generating circuit 425. The center-of- 
gravity image data generating circuit 425 outputs binary 
data (1) to one data transfer line 520j, that corresponds to 
the y-axis position Py for the center of gravity, at a 
timing that corresponds to the x-axis position Px for the 
center of gravity, thereby generating center-of-gravity 
image data P(x, y) displaying the point of the center-of- 
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gravity position (Px, Py) . 

Next, the edge extraction circuit 403 will be 
described. As shown in Fig. 6"; the edge extraction circuit 
403 is connected to the N2 data transfer lines 440j 
extending from the binarization circuit 401. 

In the present embodiment, the edge extraction circuit 
403 has a circuit configuration for performing a 3x3 masking 
operation, enabling the circuit to perform an edge 
extraction operation using data for neighboring pixels. As 
shown in Fig. 6, the edge extraction circuit 403 includes N2 
sum-of -product computing units 430j corresponding to the N2 
rows in the photodetector array 11, that is, the N2 analog- 
to-digital converters 210j . The N2 data transfer lines 440j 
from the binarization circuit 401 are connected to 
corresponding N2 sum-of-product computing units 430j, 
respectively. In addition to being connected to the data 
transfer line 440j of the corresponding row j , each sum-of- 
product computing unit 430j is also connected to the data 
transfer lines 440j_i and 440 j+ i of the corresponding rows j 
and j+1 directly above and below the corresponding row j . 

As shown in Fig. 8, nine multipliers 431(0) through 
431(8) and one adder 432 are provided in each sum-of-product 
computing unit 430j . The output from each multiplier 431 is 
connected to a corresponding input terminal of the adder 432 
Of these multipliers 431(0) - 431(8), the multipliers 431(0) 
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through 431 (2) are connected to the data transfer line 440j- 
i ; the multipliers 431(3) through 431(5) are connected to 
the data transfer line 440j; and the multipliers 431(6) 
through 431(8) are connected to the data transfer line 440j+i 
In each row j-1, j, and j+1, each multiplier 431 is capable 
of orderly transferring image data from the corresponding 
data transfer line 440 to its subsequent multiplier 431. In 
this way, as data is transferred via the multipliers 431, 
image data for a certain photodetector element 120 and for 
the eight photodetector elements surrounding that 
photodetector element 12 0 is inputted into the input 
terminals of the corresponding multipliers 431. Since no 
row j-1 exists for the sum-of-product computing unit 430i, 
corresponding to the first row (j=l) , no data transfer line 
440 is connected to the multipliers 431(0) through 431(2) in 
the sum-of-product computing unit 43 0i. Accordingly, the 
image data 0 is always inputted into the multipliers 431(0) 
through 431(2) in the sum-of-product computing unit 430i. 
Similarly, since no row j+1 exists for the sum-of-product 
computing units 430 N2 corresponding to the N2 th row (j=N2) , 
no data transfer line 440 is connected to the multipliers 
431(6) through 431(8) in the sum-of-product computing units 
430 N2 - Accordingly, image data (0) is always inputted into 
the multipliers 431(6) through 431(8) in the sum-of-product 
computing units 430 N2 . 
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The other input terminal of each of the multipliers 
431(0) through 431(8) is connected to a mask data supply- 
circuit 450. Mask data specifying details of an operation 
(in this case, mask data for performing edge extraction) are 
transferred from the mask data supply circuit 450 to the 
multipliers 431(0) through 431(8). 

In each sum-of-product computing unit 430j having this 
construction, a data transfer line 530 j is connected to the 
output terminal of the adder 432 for outputting processed 
image data D'(x, y) , where y=j . The data D' is the sum 
calculated by the adder 432. Therefore, N2 data transfer 
lines 530j are connected to the output terminal of the 
entire edge extraction circuit 403. As shown in Fig. 6, 
these N2 data transfer lines 530j are connected to a 
processed image data buffer 433. N2 data transfer lines 
540j are connected to output terminals of the processed 
image data buffer 433. The processed image data buffer 433 
stores, for a predetermined time period, processed image 
data D ' (x , y) which are received sequentially from each data 
transfer line 530j . Subsequently, the processed image data 
buffer 433 transfers this data sequentially to the signal 
converter 17 via the corresponding data transfer line 540j . 

As shown in Fig. 9, the N2 data transfer lines 500j 
extending from the raw image data buffer 460 , the N2 data 
transfer lines 520j extending from the center-of-gravity 
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image data generating circuit 425, and the N2 data transfer 
lines 540j extending from the processed image data buffer 
433 are connected to the signal converter 17 via N2 switches 
20j . 

In the signal converter 17, N2 data buffers 171j are 
provided. Each data buffer 171j is for accumulating a 
corresponding row (Nl pixels) of composite image data for a 
predetermined number of frames (n frames, for example, where 
n is a positive integer) . Each data buffer 171j is 
connected to the corresponding data transfer line 500j, 520j , 
and 540j via a corresponding switch 2 0j. Composite image 
data for each row j is formed by combining the raw image 
data D(x, y) , center-of-gravity image data P(x, y) , and 
processed image data D'(x, y) and is inputted into each 
corresponding data buffer 171j by switching each switch 20j 
at a predetermined timing. 

Further, a digital-to-analog converter 172 and a 
synchronization signal mixer 173 are connected to the data 
buffers 171j. The D/A converter 172 converts the composite 
image data, which is in the form of digital image signals, 
to analog signals. The synchronization signal mixer 173 
generates image signals at a reguired frame rate by adding a 
synchronization signal, appropriate for the frame rate of 
the monitor 18 (or a subseguent general-purpose image- 
processing device, such as a computer) , to the analog 
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signals inputted from the digital-to-analog converter 172 . 

The control circuit 15 includes a CPU, memory, and the 
like not shown in the drawings, and controls overall 
operations of the camera system 1 via the 
instruction/command bus and the signal conversion controller 
19. To achieve this, the control circuit 15 executes a 
control program described below (see Fig. 12) stored in its 
memory. The signal conversion controller 19 includes a CPU, 
memory, and the like not shown in the drawings. Based on 
instructions received from the control circuit 15, the 
signal conversion controller 19 controls the signal 
converter 17 to accumulate raw image data and processed 
image data and to perform signal conversion of this data. 
It is possible to configure the control circuit 15 and the 
signal conversion controller 19 using the same CPU. 

Next, the operations of the camera system 1 according 
to the present embodiment will be described with reference 
to Fig. 10. 

The camera system 1 of the present embodiment serves 
as an image processor by performing binarization , center of 
gravity calculation, and edge extraction operations. The 
camera system 1 overlays the processed result on the 
original image . 

First, the reset signal R is reset to enabled, all of 
the switches SW11 - SW14 in the variable capacity unit 222 



of Fig. 4 are set to ON, and all the switches SW21 - SW24 
are set to OFF. With this operation, the capacity between 
the input and output terminals of the charge amp 221 is set 
to Co- At the same time, all the switch elements 140 shown 
in Fig. 3 are set to OFF, and the horizontal scan signal Vj. 
is set to a state that does not select any photodetectors 
120. From this state, the reset signal R is set to disabled, 
and each integrating circuit 220 is controlled to start 
integration operations . 

After starting the integration operation, a horizontal 
scan signal Vi is outputted for setting, to ON, only the 
switch element 140 in the first photodetector 120i,j among 
the photodetectors in each of the N2 horizontal 
photodetector rows 110j shown in Fig. 3. When the switch 
element is thus turned ON, a charge Qi, which has been 
accumulated in the photoelectric conversion element 130 in 
response to light received until present, is outputted as a 
current signal from the light receiving portion 100. In 
this way, it is possible to read out a signal indicative of 
a raw image 901 of an object 900, shown in Fig. 9, which is 
taken by the photodetectors. The charge Qi then flows to 
the variable capacity unit 222 , which is now being set at 
capacity Co- 

Next, the internal operations of the integrating 
circuit 220 will be described with reference to Fig. 4. The 
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capacity control mechanism 240 (Fig. 3) opens switches SW12 
- SW14, and then closes the switches SW22 - SW2 4. As a 
result, the integral signal V s is outputted to have the 
following voltage value: 
V s = Q/Cx 

The integral signal V s is inputted into the comparator 
circuit 230, and is compared with the reference voltage Vre F . 
If the difference between the integral signal V s and the 
reference voltage Vre F is within the range of resolution, 
that is, lower than or equal to ±C 4 /2, then the integral 
signal V s is determined to match the reference voltage Vref- 
In this case, no further capacity control is performed and 
the integral operation ends. On the other hand, if the 
difference does not fall within the range of resolution, 
then further capacity control will be performed to continue 
the integral operation. 

For example, if V 3 > Vref, the capacity control 
mechanism 240 opens the switch SW22 and subsequently closes 
the switch SW12. As a result, the integral signal V s 
changes to have the following voltage value: 

V s = Q/ (Ci + C 2 ) 

This integral signal V s is inputted into the 
comparator circuit 230 (Fig. 3) and is compared with the 
reference voltage V REF . 

On the other hand, if V s < Vref, then the capacity 



control mechanism 240 opens both switches SW11 and SW22 and 
subsequently closes the switches SW12 and SW21. As a result, 
the integral signal V s changes to have the following voltage 
value : 

V s = Q/C 2 

This integral signal V s is outputted to the comparator 
circuit 230 to be compared with the reference voltage Vre F . 

By repeating this feedback loop among the integrating 
circuit 220, the comparator circuit 230, the capacity 
control mechanism 2 40, and back to the integrating circuit 
220, comparison operation and capacity setting operation 
(that is, control of the ON/OFF settings of the switches 
SW11 - SW14 and SW21 - SW24) are repeated until the integral 
signal V s matches the reference voltage Vref within the 
resolution range. The values of the capacity instructions 
signals Cu - Ci 4 , which indicate the ON/ OFF states of the 
switches SW11 - SW14 at the time when the integration 
operations are ended, constitute a digital signal that 
corresponds to the charge Qi , wherein the most; significant 
bit (MSB) of the digital signal is the value of Cu, while 
the least significant bit (LSB) is the value of Cu. The 
input signal is thus analog-to-digital converted into the 
digital signal, and the digital signal is outputted as a 
digital signal Dl to the processing element array 14 via the 
corresponding data transfer line 250. 



38 



As described above, the present apparatus successively 
determines the values of the respective bits in the digital 
signal Dl one bit at a time from the MSB to the LSB . 

Hence, according to the present embodiment, the 
integral signal V s is repeatedly compared with the reference 
voltage Vre F , while the capacity elements CI - C4 are 
successively turned ON one at a time. The comparison 
result is outputted as a digital signal Dl . More 
specifically, CI is first set to ON/ thereby making integral 
signal V s equal to Q/Cl . This integral signal V s is compared 
with the reference voltage Vre F . A one (1) (= Cn) is 
outputted as the MSB (most significant bit) if the integral 
signal V s is larger than the reference voltage Vre F . A zero 
(0) (= Cn) is outputted as the MSB (most significant bit) 
if the integral signal V s is smaller than the reference 
voltage . Next, C2 is set to ON, making the integral 

signal V s be equal to Q/ (CI + C2) when MSB = 1, or the 
integral signal V s be equal to Q/C2 when MSB = 0. This 
integral signal V s is compared to the reference voltage Vre F . 
A one (1) (= Ci 2 ) is outputted as the second bit if the 
integral signal V s is larger than the reference voltage Vre F . 
A zero (0) (= C12) is outputted as the second bit if the 
integral signal V s is smaller than the reference voltage Vre F 
Analog-to-digital conversion is executed by repeating the 
above-described process the number of times that is equal to 
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the number of bits required. 

When output of a digital signal equivalent to the 
photoelectric output from the first photodetector 120i,j is 
completed, the reset signal R is enabled. The reset signal 
is then disabled again, and the capacity value of the 
variable capacity unit 222-j is initialized. Subsequently, a 
horizontal scan signal V 2 is outputted for turning ON the 
switch element 140 only in the second photodetector 120 2 ,j in 
each row 110j . Then, the above-described process is 
performed again for reading the photoelectric output from 
the second photodetector 120 2 ,j and for outputting a 
corresponding digital s„ignal . Hereafter, by successively 
changing the horizontal scan signal Vi, photoelectric 
outputs are read successively from all the photodetectors 
120, and corresponding digital signals are outputted to the 
processing element array 14. 

As a result, as shown in Fig. 11(a), to each data 
transfer line 250j , outputs from all the photodetectors 
120i,j - 120 N i,j in the corresponding j -th row are outputted 
successively in this order. The output from each 

photodetector 120 i; j is outputted from its most significant 
bit (MSB) to its least significant bit (LSB) in this order. 
The four-bit raw image data D(x,y) = 901 (shown in Fig. 10) 
thus outputted from each photodetector 120i,j is transmitted 
not only to the binarization processing circuit 401 but also 
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to the raw image data buffer 460, in which the four-bit raw 
image data D(x,y) is stored for the predetermined period of 
time . 

According to the configuration of the analog-to- 
digital converter array 13 of the present embodiment, the 
analog-to-digital conversion rate per pixel is about 1 y, 
s/bit. For example, if 128 photodetectors 120 exist in each 
row, it takes 1.024 ms (= 8 m s x 12 8) to convert input 
analog signals from all the 128 photodetectors 120 in one 
row into 8-bit (12 8 gradation) digital signals, for example. 
In other words, it is possible to acquire each frame image 
within about one millisecond. It is therefore possible to 
obtain successive frames' worth of images at a high frame 
rate of about 1,000 frames/second. 

Next, the operations of the image processing unit 14 
will be described with reference to Fig. 1 and Figs. 6 
through 10. 

First, the binarization process performed by the 
binarization circuit 401 will be described with reference to 
Fig. 6. 

Specifically, each comparator 405j receives image data 
901 (Fig. 10) from the corresponding analog-to-digital 
converter 210-;. As described above, the image data 901 is a 
series of digital signals inputted beginning from its MSB. 
Each comparator 405j also receives a comparison signal, to 
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be used for comparison, from the control circuit 15. The 
comparison signal is a series of signals inputted beginning 
from the MSB. The comparator 405j performs sequential 
comparisons on the thus inputted data. The comparator 405 
outputs one (1) when the image signal matches or is greater 
than the comparison signal, and zero (0) when the image 
signal is smaller than the comparison signal. Accordingly, 
each comparator 405j outputs 1-bit binary image data via the 
corresponding data transfer line 440 3 . As a result, a 
binary image 902 corresponding to the raw image data 901 is 
obtained as shown in Fig. 10. This binary image data is 
transmitted to the center-of-gravity calculating circuit 402 
and edge extraction circuit 403 via the data transfer lines 
440j . 

Subsequently, center-of-gravity data and an edge 
extraction image are obtained based on the binarized image 
data . 

The center-of-gravity data is calculated as described 
below by the center-of-gravity calculating circuit 402 
having the construction shown in Fig. 7. 

Assuming that the binary image data inputted to the 
circuit 402 is D(x, y) , the center-of-gravity coordinates 
(Px, Py) for the X and Y directions can be determined 
according to the following equations: 
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The center-of-gravity detecting circuit 409 in the 
center-of— gravity calculating circuit 402 calculates a 
common denominator and the respective numerators in the 
equations (a) and (b) in parallel processes. 

In other words, each adder 410j is inputted with 
binarized image data for all pixels in the corresponding row 
j via the corresponding data transfer line 440j . 
Accordingly, each adder 410j adds binary image data for all 
the Nl pixels in the corresponding row j . Each multiplier 
411j multiplies this sum by the corresponding row position 
data y received from the row position data supply circuit 
412. The adder 413 obtains a value for the numerator of 
equation (b) by adding these products for all N2 lines. 

All the data transfer lines 440j are also connected to 
the adder 415. Hence, the adder 415 adds the binary image 
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data in each column i for all N2 rows. The adder 418 
further adds the sums of each column i for all the Nl 
columns to obtain a denominator for each equation (a) and 
(b) . 

The binary image data added for each column i by the 
adder 415 is also sent to the multiplier 416. The 
multiplier 416 multiplies a corresponding column position 
data x, received from the column position data supply 
circuit 412, by the sum for each column i and outputs this 
result to the adder 419. The adder 419 obtains a numerator 
for equation (a) by adding the results obtained by the 
multiplier 416 for all Nl columns. 

The divider 414 obtains an output corresponding to Py 
by dividing the total sum of image data weighted by the row 
position data y by the total sum of image data outputted 
from the adder 418. The divider 420 obtains an output 
corresponding to Px by dividing the total sum of image data 
weighted by the column position data x by the total sum of 
image data outputted from the adder 418. 

The center-of-gravity data (Px, Py) is transferred to 
the control circuit 15 via the data transfer lines 510x, 
510y, and the instruction/command bus 16. 

After receiving the center-of-gravity data (Px, Py) 
via the data transfer lines 510x and 510y, the center-of- 
gravity image data generating circuit 425 generates center- 



of-gravity image data P(x, y) in which only the point at the 
center-of-gravity position (Px, Py) is set to one and all 
the remaining pixels (x, y) , wherein l^x^Nl, x^Px, l^y^ 
N2 , and y^Py, are set to 0. This data is outputted via the 
data transfer lines 520-j. In other words, the data 0 is 
continuously outputted to all the data transfer lines 520j 
not corresponding to Py (that is, j # Py) . The data 1 is 
outputted only to the data transfer line 520j corresponding 
to Py (that is, j=Py) at a timing corresponding to Px . The 
data 0 is outputted to the data transfer line 520j 
corresponding to Py (that is, j=Py) at all the remaining 
timings. In this way, the center-of-gravity image data P(x, 
y) is generated. 

The thus produced center-of-gravity image data P(x, y) 
is an image indicating a point only at the center-of-gravity 
position, such as an image 904 in Fig. 10. In the image 904, 
the intersecting point of an X within a circle denotes the 
center-of-gravity position. 

Next, the edge extraction process will be described. 
The edge extraction process is performed by the edge 
extraction circuit 403. The edge extraction circuit 403 is 
provided with N2 3x3 sum-of-product computing units 430j 
each having the construction shown in Fig. 8. 

Each 3x3 sum-of-product computing unit 430j performs 
the image process expressed by the following eguation: 
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D' (i,j) 

= mask (0) xD (i-1 , j-1) + mask (1) xD (i, j-1) + mask (2) xD (i+1 , j-1) 
+ mask(3)xD (i-1, j ) + mask (4) xD (i , j ) + mask (5) xD (i+1 , j ) 
5 + mask (6) xD (i-1 , j+1) + mask (7 ) xD ( i , j +1 ) + mask (8 ) xD (i+1 , j +1 ) 

... (c) 

O In this equation, D(i, j), equivalent to D(x, y) , is 

CO the inputted binary image data. More specifically, D(i-1, 

HO j), D(i, j), and D(i+1, j) are binary data for three 
111 neighboring pixels that are transmitted via the data 

O transfer line 440j . Similarly, D(i-1, j-1), D(i, j-1), and 

D(i+1, j-1) are binary data for three neighboring pixels 
O that are transmitted via the data transfer line 440-j_i. In 

15 addition, D(i-1, j+1), D(i, j+1), and D(i+1, j+1) are binary 
data for three neighboring pixels transmitted via the data 
transfer line 440j+i. 

D' (x, y) is the image data outputted as a result of 
the process. The "mask" is masking data supplied from the 
20 mask data supplying circuit 405. 

In the edge extraction process, a Laplacian operation 
is performed with the mask values set as shown below. 
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mask - {1,1,1,1,-8,1,1,1,1} 



•(d) 



The binary image data D(x, y) is transferred into the 
3x3 sura-of -product computing unit 430 via the data transfer 
lines 440. As a result, image data D(i±l, j±l) for 
neighboring 3x3 pixels is inputted into the multipliers 431. 
At the same time, the mask data supply circuit 450 inputs 
the mask data shown in equation (d) into each multiplier 431 
That is, a value of one (1) is inputted into the multipliers 
431(0) through 431(3) and multipliers 431(5) through 431(8), 
while a -8 value is inputted into the multiplier 431(4). 
The result of multiplying the image data with the inputted 
mask data is transmitted to and added in the adder 432 . The 
resulting sum data D'(x, y) , that is, the edge detection 
data is outputted. This outputted edge extraction data is 
equivalent to an edge extraction image 903, shown in Fig. 10. 

Hence, the edge extraction results D'(i, j) obtained 
from each of the 3x3 sum-of-product computing units 430-, is 
outputted to the corresponding data transfer line 530j . 
After being stored for a predetermined period of time in the 
processed image data buffer 433, the data is outputted via 
the corresponding data transfer line 540j . 
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The above operation uses parallel processing for 
processing all the lines simultaneously, thereby reducing 
processing time and increasing the operation speed. 
Accordingly, images acguired at a high-speed frame rate, 
which is determined by the analog- to-digital conversion rate, 
can be processed in real-time. 

Next, the process of overlaying the processed images 
and raw image data will be described with reference to Figs. 
9 through 11 (c) . 

As described above, a signal for one row of the raw 
image data 901 is outputted for each row j. As shown in Fig. 
11 (a) , the one-row signal is outputted sequentially as a bit 
serial signal beginning from its MSB. Data for each pixel 
is composed of four bits, for example. A signal-less 
interval is inserted between successive sets of pixel data 
in order to reset the amplifiers and the analog-to-digital 
converters. As shown in Fig. 9, the switch 20 j 

corresponding to each row j is controlled to switch the 
input of signals to be sent to the corresponding data buffer 
171 3 among: the data transfer line 520j from the image- 
processing unit 14, the data transfer line 540-j from the 
image-processing unit 14, and the data transfer line 500j 
from the analog-to-digital converter array 13 that 
circumvents the image-processing unit 14. As shown in Fig. 
11(b), the image processing results (in this case, the 
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center-of-gravity image data and edge extraction image data) 
are inserted during the signal-less interval between 
successive pixel data for the raw image. This process 
generates six bits of data from the four bits of raw image 
data and the additional two bits of processed image data 
(one bit for center-of-gravity image data and one bit for 
edge extraction image data) . 

Here, the switch 20j corresponding to each row j is 
switched seguentially between the data transfer line 500j 
from the analog-to-digital converter array 13 (raw image 
data buffer 450) , the data transfer line 520j from the 
center-of-gravity image data generating circuit 425, and the 
data transfer line 540j from the edge extraction circuit 403 
(processed image data buffer 433) . This switching process 
combines the raw image data, center-of-gravity image data, 
and edge extraction image data. 

In this way, the one-bit center-of-gravity image data 
(0 or 1) and the one-bit edge extraction image data (0 or 1) 
obtained as processed results are added before or after the 
four bits of raw image data for each pixel i in every row j . 
It is noted that because a certain amount of time is 
reguired for the center-of-gravity calculating circuit 402 
to complete the center-of-gravity process, the raw image 
data is stored temporarily in the raw image data buffer 450 
and the image data from the edge extraction process is 
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stored temporarily in the processed image data buffer 433. 
These raw image data and edge extraction image data are 
outputted from the corresponding data buffers 450 and 433 in 
synchronization with the timing in which the center-of- 
gravity image data generating circuit 425 generates the 
center-of-gravity image data. 

As shown in Fig. 11(b) , for example, one-bit signals 
for the center-of-gravity data and one-bit signals for edge 
extraction image are inserted in this order following the 
four bits of raw image data. As a result, a composite image 
905 is generated as shown in Fig. 10. This composite image 
9 05 emphasizes the edges in the raw image and displays the 
center-of-gravity position. 

It is unnecessary to provide the data buffer 450 or 
433 in this construction. In such a case, the center-of- 
gravity image data can be inserted in the position of the 
corresponding pixel at a timing one frame later than that of 
the raw image data and edge extraction image data. 

When it is necessary to maintain a four-bit length for 
the output image data, due to the restrictions by the 
display device and processing devices in later stages, the 
processed results can be inserted in place of the LSB or MSB 
of the raw image data. For example, as shown in Fig. 11(c), 
the one bit signal for center-of-gravity data and the one 
bit signal for edge extraction image can be inserted in this 
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order in place of the two low-order bits in the raw image. 
This process also generates a composite image 905 
emphasizing edges of the raw image and displaying the 
center-of-gravity position as shown in Fig. 10. 

The composite image 905 thus obtained for each row j 
in this way is stored in the corresponding data buffer 171j. 
Each data buffer 171j has a capacity suited to storing the 
number of bits in the composite image 905. When the 
composite image 905 is formed of six bits for each pixel, as 
in the example of Fig. 11(b), for example, each data buffer 
171j has a capacity of 6 bits x Nl (number of pixels in each 
row j) x n (number of frames) . When the composite image 905 
is formed of four bits for each pixel, as in the example of 
Fig. 11 (c) , each data buffer 171j has a capacity of 4 (bits) 
x Nl (number of pixels in each row j) x n (number of frames) 
Accordingly, all N2 data buffers 171j can store the 
composite image 905 for n frames. 

The camera system 1 of the present embodiment 
repeatedly obtains image data 901 for a plurality of 
consecutive frames at a high frame rate which is determined 
by the analog- to-digital conversion rate of the analog-to- 
digital converter array 13. The camera system 1 then 
repeatedly generates and stores a composite image 905 for 
each frame. The camera system 1 converts some composite 
images, that have been acquired at desired timings, into a 



desired frame rate, and displays the images on the monitor 
18. 

This desired time, for example, may be determined as 
the time when the center-of-gravity position (Px, Py) 
determined by the center-of-gravity detecting circuit 409 
matches a desired set position. Each time an image for one 
frame is obtained, the control circuit 15 receives data of 
the center-of-gravity position (Px, Py) from the center-of- 
gravity detecting circuit 409 and determines whether this 
position matches the predetermined set value (X, Y) . When 
the position does not match, the signal conversion 
controller 19 is controlled to store the composite image 
data for the present frame in the data buffer 171. However, 
when the center-of-gravity position (Px, Py) for the present 
frame is determined to match the set position (X, Y) , the 
control circuit 15 outputs an instruction to the signal 
conversion controller 19 to perform a monitor display 
process. When commanded to perform a display process, the 
signal conversion controller 19 controls the signal 
converter 17 to read composite image data for n frames 
already stored in the data buffer 171, reading in order from 
the oldest frame, convert the data to signals suitable for 
the monitor display, and display the images on the monitor 
18. The same process is performed to display the present 
frame as well. The same process is also performed to 
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display composite image data for the next n frames in 
sequence. Based on commands from the signal conversion 
controller 19, the signal converter 17 converts the overlaid 
image signals to image signals appropriate for displaying on 
the monitor 18, and displays those image signals on the 
monitor 18. 

The camera system 1 of the present embodiment performs 
the above-described process following the flowchart shown in 
Fig. 12 under the control of the control circuit 15. 

First, in S10, the control circuit 15 controls the 
photodetector array 11 and the analog-to-digital converter 
array 13 to acquire raw image data (x, y) for one frame, 
wherein l^x^Nl and l^y^sN2. The raw image data D(x, y) is 
transferred via the data transfer lines 250j to the image- 
processing unit 14 and stored in the raw image data buffer 
460 . 

In S20, the control circuit 15 controls the image- 
processing unit 14 to perform image processing. First, the 
binarization circuit 401 is controlled to binarize the raw 
image data received from the data transfer lines 250j to 
thereby generate binary image data. Next, the center-of- 
gravity calculating circuit 402 is controlled to calculate 
the center-of-gravity position (Px, Py) in the present frame 
based on the binary image data. The results of this 
calculation are transmitted both to the control circuit 15 
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and to the center-of-gravity image data generating circuit 
425. The center-of-gravity image data generating circuit 
4 25 creates center-of-gravity image data. Further, the edge 
extraction circuit 403 is controlled to generate edge 
extraction image data based on the binary image data. The 
edge extraction image data is stored in the processed image 
data buffer 433. 

In S30, the control circuit 15 compares the center-of- 
gravity position (Px, Py) received for the present frame 
with the set data (X, Y) . If the center-of-gravity position 
(Px, Py) does not match the set data (X, Y) (No in S40) , 
then the control circuit 15 controls the signal conversion 
controller 19 to generate the composite image data for the 
current frame and to store the composite image data in the 
data buffers 171 of the signal converter 17. More 
specifically, the control circuit 15 controls the switches 
20j to combine the raw image data in the raw image data 
buffer 460, the center-of-gravity image data from the 
center-of-gravity image data generating circuit 425, and the 
edge extraction image data from the processed image data 
buffer 433, and to store this composite image in the data 
buffers 171. The next frame is then acquired and processed 
beginning from S10. It is noted that when n frames of 
composite image data have already been accumulated in the 
data buffers 171, composite image data for the oldest frame 



is discarded and composite image data for the current frame 
is stored in the data buffer 171. 

On the other hand, if the center-of -gravity position 
(Px, Py) for the current frame matches the set data {X, Y) 
(Yes in S40) , the control circuit 15 controls the signal 
conversion controller 19 to initiate a monitor display 
process in S60. After the control circuit 15 sends the 
instruction to the signal conversion controller 19, the 
process returns to S10 and the control circuit 15 begins 
acquiring and processing the next frame. 

After receiving the instruction to initiate the 
monitor display process, the signal conversion controller 19 
begins controlling the signal converter 17 in S70 to display 
composite image data already stored in the data buffers 171, 
composite image data for the current frame, and composite 
image data for subsequent n frames in order. More 
specifically, the signal conversion controller 19 reads in 
order of oldest to newest all frames (n frames) currently 
stored in the data buffer 171. The signal conversion 
controller 19 converts these frames to analog signals using 
the digital-to-analog converter 172, combines the analog 
signals with a synchronization signal in the synchronization 
signal mixer 173, and displays the resulting signal on the 
monitor 18. The signal conversion controller 19 also 
performs the same process to display the composite image 
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data for the current frame and for the subsequently received 
n frames . 

The control process described above is used for 
displaying images before and after a specific instant. 
5 However, if it is only necessary to display images for the 
frame of the specific instant and frames thereafter, there 
is no need to accumulate images in S50. In this case, 
g images need to be accumulated and displayed only for the 

jn frame at which a Yes is determined in S40 and several frames 

yglO thereafter. 

i~y 

a As described above, according to the present 

P 

yi embodiment, the analog-to-digital converter array 13 

fU 

\j includes one analog-to-digital converter 210 for each row of 

o 

py photodetector elements 120 in the photodetector array 11. 

15 The image-processing unit 14 includes the plurality of 
processing circuits 400 for performing high-speed image 
processing. The signal converter 17 combines the output 
signals from the analog-to-digital converter array 13 with 
output signals from the image-processing unit 14. Under 

20 control of the control circuit 15 and the signal conversion 
controller 19, the signal converter 17 also downconverts the 
composite signal at an important timing to a frame rate 
suitable for display on the monitor 18 and subsequently 
displays the signal on the monitor 18. Accordingly, it is 

25 possible to extract only the necessary frames from among 
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images that are acquired and processed at a high frame rate 
and to convert and display only those signals. 

In apparatus using conventional technology no. 4, the 
frame rate of image sensors used to display the images is 
limited to 30 milliseconds. Moreover, it is impossible to 
perform high-speed image processing because image 
transmission and image processing have to be conducted in 
conformity to the image display timings, as shown in Fig. 
13(b). In contrast, the apparatus of the present embodiment 
performs image transfer and image processing at the fast 
frame rate of 1,000 milliseconds. This apparatus stores 
video images of arbitrary timings in the data buffers 171, 
then generates and outputs a video signal conforming to the 
frame rate of the monitor 18 using the digital-to-analog 
converter 172 and the synchronization signal mixer 173. 
Accordingly, the apparatus of the present embodiment can 
extract video images at certain timings and display those 
images while slowing down the display speed thereof as shown 
in the timing chart of Fig. 13 (a) . 

Figs. 14(a) through 14(c) illustrate an example of how 
the camera system 1 according to the preferred embodiment 
can extract images at the instant in which a chip 53 is 
mounted on a substrate 50 using a device 54. Here, chips 51 
and 52 are already mounted on the substrate 50. Fig. 14(b) 
is an image of the instant at which the chip 53 is mounted, 
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while the images in Figs. 14(a) and 14(c) are acquired 
before and after the instant in Fig. 14(b) . 

In order to extract an image at the moment when the 
chip 53 is mounted, the camera system 1 of the present 
5 embodiment performs a special characteristic extracting 
process to extract the chip 53, which is the target image, 
from the inputted image. The camera system 1 outputs the 
O image when the target is mounted at a predetermined position 

00 on the substrate. 

M=10 More specifically, the input image D(x, y) is first 

ify transferred to the image-processing unit 14. The image- 

O processing unit 14 performs the center of gravity 

y 

Hi calculation processes and the edge extraction processes to 

O generate processed images. The positions of "the chip 53 are 

ru 

15 found quickly (at a fast frame rate) based on the results of 

the center-of-gravity calculation. At a timing, when the 
chip 53 reaches the predetermined position on the substrate 
50, the signal conversion controller 19 instructs the signal 
converter 17 to convert image signals. Video images before 

20 and after the image at this timing (the image shown in Fig. 
14(b)) are displayed on the monitor 18. 

With this function, the apparatus can extract a series 
of images containing the most essential data, such as that 
just before and after the instant at which a drill contacts 

25 a machining workpiece or the instant at which a bat meets a 
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ball, for example, and output these images to a monitor. 
Because most conventional image processings can be 
implemented in hardware in a similar manner as described 
above, it is possible to process images of fast moving 
obj ects . 

Furthermore, the edge extraction circuit 403 that 
performs a 3x3 mask operation can perform various operations 
in addition to edge extraction by varying the mask data used 
in equation (c) . For example, the mask {1, 1, 1, 1, 8, 1, 1, 
1, 1} can be used for a smoothing process; the mask {-1, 0, 
1, -1, 0, 1, -1, 0, 1} can be used for vertical line 
detection; and the mask {-1, -1, -1, 0, 0, 0, 1, 1, 1} can 
be used for horizontal line detection. 

In the embodiment described above, the composite image 
is displayed by overlaying the raw image and the processed 
images. However, the display screen could be split in half, 
simultaneously displaying the raw image in one half and the 
processed image (s) in the other. For example, in order to 
display the raw image and the processed image simultaneously 
in the left and right halves of the monitor, when 
transferring signals to the data buffers 171 shown in Fig. 9, 
the switches 20 can be changed to transfer signals from the 
analog-to-digital converter array 13 to a buffer region 
corresponding to the pixels displaying the raw image and to 
transfer signals from the image-processing unit 14 to a 
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buffer region corresponding to the pixels displaying the 
processed image. 

By switching each switch 2 0 at an arbitrary timing, it 
is possible to output only the raw image, to output only one 
5 processed image (only the center-of-gravity image or only 
the edge extraction image) , or to output only the composite 
processed image (the composite image including the center- 

£3 

y of-gravity image and the edge extraction image) . Hence, 

'f=f images can be outputted in a user's desired format. 

j|3l0 The binarization circuit 401 can be configured to pass 

m the raw image data unchanged and to insert its binarized 

y 

y= result after the LSB of this raw image data. Similarly, the 

Rj 

2 edge extraction circuit 403 can be configured to allow the 

1=1 

IU raw image data to pass unchanged. In this case, it becomes 

15 unnecessary to provide the data transfer lines 500j that 
lead from the analog-to-digital converter array 13 to the 
switches 20 circumventing the image-processing unit 14. 

More specifically, when binarizing the raw image data, 
each of the bit comparators 405^ in the binarization circuit 
20 401 adds the binarized data to the raw image data and 
outputs the result to the corresponding data transfer line 
440j , as shown in Fig. 15. In addition, as indicated by the 
dotted line in Fig. 8, a data transmission line 550 j for 
outputting raw image data D(x, y) for each row j is 
25 connected to the output terminal of the multiplier 431(5) in 
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each sum-of-product computing unit 430j . In this case, all 
N2 data transfer lines 530j and all N2 data transmission 
lines 550j are connected to the output terminals of the N2 
sum-of-product computing units 430j , as shown in Fig. 15. A 
raw image data buffer (a raw image data buffer 435 , in this 
example) is connected to all the N2 data transmission lines 
550j. N2 data transfer lines 560j are connected to the 
output terminal of the raw image data buffer 435. This 
construction eliminates the need to provide an extended 
branch on the data transfer lines 250j, from the analog-to- 
digital converter array 13, in order to circumvent the 
image-processing unit 14. As shown in Fig. 16, therefore, 
composite image data can be generated by switching the 
switch 2 0j corresponding to each row j in order to connect 
the corresponding data transfer line 560j from the raw image 
data buffer 435, the corresponding data transfer line 540j 
from the processed image data buffer 433, and the 
corresponding data transfer line 520j from the center-of- 
gravity image data generating circuit 425 to the 
corresponding data buffer 171j . 

In the above description, the center-of-gravity 
position is determined from the binarized image. However, a 
more precise center-of-gravity position can be determined 
using a similar calculation on the raw image, although the 
processing load is increased. Similarly the edge extraction 



process can be performed based on the raw image data. 

A computer or other processing device can be connected 
to the signal converter 17 in place of the monitor 18. In 
this case, only composite image data of the extracted frames 
is outputted to the processing device at a desired frame 
rate. Accordingly, the processing device can easily perform 
required operations on the necessary images. 

The object of the present embodiment is to provide an 
image sensor device, which extracts an image at a desired 
timing based on high-speed processed results, which combines 
the processed results with a raw image, and which outputs 
the combined results or displays the combined results for 
monitoring. The necessary processing speed is determined, 
for the factory automation robot, for example, dependently 
on the moving speed of an object and on the actuator speed 
of the robot (1 - 10 ms) . According to the present 
embodiment, the processing speed is determined by the speed 
of the analog-to-digital conversion performed by the analog- 
to-digital converters 210. The number of bits in the 
outputs from the analog-to-digital converters 210 can be 
varied. Accordingly, if a higher-speed processing is 
required, it is possible to increase the processing speed by 
increasing the analog-to-digital conversion speed by 
reducing the number of bits. For example, if it takes 10 n 
s to perform conversion into eight bites, it is possible to 
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make the conversion rate two times as high as the original 
conversion rate by reducing the data length into four-bit 
length from the eight-bit length. 

In the embodiment described above, each data buffer 
171 in the signal converter 17 is designed for buffering 
data of a plurality of frames. However, this construction 
can be simplified by omitting the data buffer 171, provided 
that acquiring still images at a specified timing is 
sufficient . 

In the above-described system, in order to perform 
processing at a higher frame rate, it is possible to 
increase the frame rate by increasing the total number of 
pixels, from which data is transmitted per a unit time, by 
decreasing the number of tones (gradation) of conversion 
attained by the analog-to-digital converters 210. It is 
possible to increase the frame rate by making the data 
amount per one pixel from 8 bits (256 tones) through one bit 
(binary) . For example, it is possible to increase the frame 
rate eight times as high as the original frame rate by 
changing the original data amount of 8 bits per pixel into 
one bit per pixel. More specifically, in the analog-to- 
digital converter shown in Fig. 3, comparison operation is 
performed for one pixel from its most significant bit to a 
lower bit. When the conversion has finished to a desired 
bit number, the conversion operation is reset. Then, the 
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conversion operation is proceeded to the next pixel. It is 
therefore possible to attain the analog-to-digital 
conversion with an arbitrary gradation without changing the 
hardware construction. According to the present system, bit 
serial transfer method is employed in order to attain a 
parallel processing with using a small number of data lines. 
In other words, the outputs from each analog-to-digital 
converter are transferred via a corresponding single data 
line from the most significant bit. It is possible to 
easily cope with the change in the total number of bits by 
merely performing processing operation onto data of the 
effective bit number. 

Another method for processing at a high frame rate is 
to perform calculations at a reduced resolution to reduce 
the amount of data. This can be achieved by simultaneously 
reading signals from several neighboring photodetector 
elements 120 in the photodetector array 11 shown in Figs. 1 
and 3. By reading a 128x128 pixel image at 64x128 pixels 
(simultaneously reading every two adjacent pixels in the 
horizontal direction) or at 32x128 pixels (simultaneously 
reading every four adjacent pixels in the horizontal 
direction) , it is possible to reduce the amount of image 
data by 25% or 50%, thereby improving the processing speed 
by two times or four times. This procedure for achieving 
high-speed processing by taking the sum of electric currents 
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from several pixels is called "binning." 

The above two functions can be combined to perform 
adaptive image processing on an object. For example, the 
object is tracked at a high frame rate during normal 
monitoring, while acguiring images at a small number of 
analog tones and a rough resolution. When approaching a 
desired timing, the process is switched to acguire multiple 
toned high-resolution images for more detailed inspection. 

Next, a camera system for high-speed image processing 
according to a second embodiment of the present invention 
will be described with reference to Figs. 17 through 20. 

Fig. 17 shows the construction of the camera system 1 
according to the second embodiment of the present invention. 
Parts and components the same or similar to those in the 
camera system for high-speed image processing of the first 
embodiment are designated by the same reference numerals to 
avoid duplicating description. 

The present embodiment differs from the first 
embodiment in that the camera system of the present 
embodiment is provided with an image pre-processing circuit 
1401 in place of the binarization circuit 401. The image 
pre-processing circuit 1401 is for reducing noise. The 
camera system of the present embodiment also includes an 
image post-processing circuit 1403 that is constructed from 
the center-of-gravity calculating circuit 402 and edge 
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extraction circuit 403. In addition, an image comparing 
circuit 1402 is provided between the image post-processing 
circuit 1403 and image pre-processing circuit 1401. 

In the image-processing unit 14 of the present 
embodiment, a data controller 1404 and a data buffer 1405 
are connected to the processing element array that is 
constructed from the processing circuits 1401 through 1403. 
Here, the data buffer 1405 is connected to the image 
comparing circuit 1402 , and stores background image data to 
be supplied to the image comparing circuit 1402 . The data 
controller 1404 controls the transfer of background image 
data from the data buffer 1405 to the image comparing 
circuit 1402. 

Accordingly, the image-processing unit 14 in the 
present embodiment includes the image pre-processing circuit 

1401, image comparing circuit 1402, image post-processing 
circuit 1403, data controller 1404, and data buffer 1405. 

The control circuit 15 in the present embodiment is 
connected to the data buffer 1405, data controller 1404, 
image pre-processing circuit 1401, image comparing circuit 

1402, and image post-processing circuit 1403 via the 
instruction/ command bus 16. The control circuit 15 controls 
each of these components . 

Another difference from the first embodiment is that 
the second embodiment has no branch portion connected to the 



N2 data transfer lines 250j leading from the analog-to- 
digital converter array 13 to circumvent the image- 
processing unit 14. The second embodiment is also not 
provided with the raw image data buffer 460. 

Accordingly, the camera system for high-speed image 
processing of the second embodiment performs noise reduction 
as a pre-process, background removal as a comparison process, 
and edge extraction and center-of-gravity calculations as 
post-processes . 

Fig. 18 is a block diagram showing the overall 
construction of the image pre-processing circuit 1401, image 
comparing circuit 1402, and image post-processing circuit 
1403. 

A plurality of bit comparators 1407-j are provided in 
the image pre-processing circuit 1401 for performing noise 
reduction, one for each row j in the photodetector array 11. 
In other words, N2 bit comparators 1407j are provided in the 
image pre-processing circuit 1401. One input terminal of 
each bit comparator 1407j is connected to the data transfer 
line 250-j from the corresponding analog-to-digital converter 
210j. The other input terminal of each bit comparator 1407j 
is connected to the control circuit 15 via the 
instruction/command bus 16. A data transfer line 1440-j is 
connected to the output terminal of each bit comparator 
1407-j. Hence, N2 data transfer lines 1440-j are outputted 
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from the image pre-processing circuit 1401. 

The image comparing circuit 1402 is configured of a 
plurality of subtracters 1408, one for each row j in the 
photodetector array 11. Hence, N2 subtracters 1408j are 
provided in the image comparing circuit 1402 . The positive 
input terminal of each subtractor 1408-j receives, via the 
corresponding data transfer line 1440j, signals which have 
been processed by a corresponding bit comparator 1407j in 
the image pre-processing circuit 1401. The negative input 
terminal of each subtractor 1408j receives signals 
indicating a predetermined background image from the data 
buffer 1405. The output terminal of each subtractor 1408j 
is connected to the corresponding data transfer line 440j . 
Accordingly, N2 data transfer lines 440j are outputted from 
the overall image comparing circuit 1402. 

The image post-processing circuit 1403 includes the 
center-of-gravity calculating circuit 402 and the edge 
extraction circuit 403. Similar to the first embodiment, 
the center-of-gravity calculating circuit 402 includes the 
center-of-gravity detecting circuit 409 and the center-of- 
gravity image data generating circuit 42 5. Further, the 
edge extraction circuit 403 includes a plurality of sum-of- 
product computing units 430-j, which are provided in one-to- 
one correspondence with the rows j in the photodetector 
array 11 and each of which is for performing a 3x3 mask 
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operation. Further, the center-of -gravity detecting circuit 
409 and edge extraction circuit 403 are connected in 
parallel with each other to the N2 data transfer lines 440 jf 
which extend from the image comparing circuit 402 . The 
center-of-gravity detecting circuit 409 and edge extraction 
circuit 403 are provided to perform a center-of-gravity 
calculation and edge extraction process on the raw image 
data, from which noise and background image has been removed 

As in the first embodiment, N2 data transfer lines 
520j are connected to the output terminal of the center-of- 
gravity image data generating circuit 425. The processed 
image data buffer 433 is connected to the edge extraction 
circuit 403 via the N2 data transfer lines 530-,. Also, N2 
data transfer lines 540j extend out from the processed image 
data buffer 433. 

As shown in Fig. 19, the switch 20j corresponding to 
each row j switchinly connects the corresponding data 
transfer line 520j and the corresponding data transfer line 
540j to the corresponding data buffer 171j . 

According to the camera system 1 of the present 
embodiment having the construction described above, as shown 
in Fig. 20, a pre-process is performed on the raw image data 
acquired by the photodetector array 11 to reduce noise. 
Subsequently, a comparison operation is performed to remove 
the background. Then post-processes are conducted, 



69 



including a center-of-gravity detection and edge extraction. 
Through these processes, it is possible to display the 
processed results by overlaying the center-of-gravity image 
and edge extraction image . 

Next, the operations of the camera system 1 according 
to the present embodiment will be described with reference 
to Fig. 20. 

A raw image 1901 of an object 1900 taken by the 
photodetector array 11 and the analog-to-digital converter 
array 13 includes a noise component. Therefore, the image 
pre-processing circuit 1401 performs noise reduction on the 
image. That is, the bit comparator 1407j for each row j 
performs a comparison between the image signal and a 
comparison signal, which is a predetermined threshold value. 
The bit comparator 1407j outputs the image signal as is (if 
the raw image signal is four bits, for example, the four bit 
raw image signal is outputted without change) only when the 
image signal matches or is greater than the comparison 
signal, and outputs a zero (for example, the four-bit image 
signal (0 0 0 0) when the raw image signal is four bits) 
when smaller. With this process, raw image signals that are 
less than the predetermined value are eliminated as noise. 
The result of this process is a pre-processed (noise- 
reduced) image 1902 of the raw image 1901, as shown in Fig. 
20 . 
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Next, the image comparing circuit 1402 eliminates the 
background of the pre-processed image 1902 (the raw image 
data, from which noise has been removed) . Image data for a 
background image 1903 is prestored in the data buffer 1405. 
The data controller 1404 controls transmission of the 
background image data from the data buffer 1405 to each 
subtractor 1408-j. In other words, the data controller 1404 
transfers background image data for each pixel (x, y) from 
the data buffer 1405 to the corresponding subtractor 1408j 
at the same time noise-reduced image data for the same pixel 

(x, y) is inputted into the corresponding subtractor 1408-j 

(here, j=y) . 

Each subtractor 1408-j generates a noise/background- 
removed image 1904 after removing the background image 1903 
from the pre-processed image 1902 by subtracting image 
signals for the background image 1903 from image signals for 
the pre-processed image 1902. The noise/background-removed 
image 1904 is outputted to the image post-processing circuit 
1403 via corresponding data transfer lines 440j . 

Image signals for the noise/background-removed image 
1904 are transferred to the center-of-gravity calculating 
circuit 402 and edge extraction circuit 403 in the image 
post-processing circuit 1403. The center-of-gravity 

calculating circuit 402 performs the same operation 
described in the first embodiment. Accordingly, the center- 



of-gravity position is determined by the center-of-gravity 
detecting circuit 409 and a center-of-gravity image 1905 is 
generated by the center-of-gravity image data generating 
circuit 425. The edge extraction circuit 1403 generates an 
5 edge extraction image 1906. 

As shown in Fig. 19, the switch 20j corresponding to 

M= each row j is switched between the corresponding data 

===== 

O transfer line 520j and the corresponding data transfer line 

CO 

=0 540j. By doing so, the center-of-gravity image 1905 and 

** 

yQlO edge extraction image 1906 are combined to generate a 

ru 

== composite image 1907. The composite image 1907 is stored in 

O 

W the corresponding data buffer 171j . 

m 

%1 As in the first embodiment, the control circuit 15 in 

P 

fU the present embodiment controls the entire apparatus to 

15 perform the process of the flowchart shown in Fig. 12. When 
the center-of-gravity position (Px, Py) determined by the 
center-of-gravity detecting circuit 409 in a certain frame 
matches a predetermined set position (Yes in S40) , it is 
determined that the time reaches a display timing. 
20 Accordingly, the composite images 1907 for successive frames, 
including the previous n frames, the current frame, and the 
following n frames, are converted to a frame rate 
appropriate for the monitor 18 and displayed thereon. 

As described above, the image-processing unit 14 
2 5 according to the second embodiment includes the image pre- 
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processing circuit 1401, image comparing circuit 1402, image 
post-processing circuit 1403, and data buffer 1405 for 
performing high-speed image processing. The signal 

converter 17 combines signals outputted from the image post- 
5 processing circuit 1403. Under control of the control 
circuit 15 and signal conversion controller 19, the signal 
converter 17 downconverts the composite signal at an 

O important timing into a frame rate appropriate for display 

O 

m on the monitor 18, and then displays the image on the 

*y 

M=10 monitor 18. 

nj In the present embodiment, a background image is 

O stored in the data buffer 1405, and the image comparing 

111 

ill circuit 1402 generates an output image by subtracting the 

N 

Q background image from the picked up image. Therefore, 

fU 

15 images minus their backgrounds can be outputted in real-time. 

As in the variation of the first embodiment, the sum- 



of-product computing units 43 0j of the present embodiment 
also can be configured to output the unaltered input image 
data, as shown by the dotted line in Fig. 8. In this case, 

2 0 the sum-of-product computing units 430j output the unchanged 
noise/background-removed image 1904, which has had the noise 
and background removed. The processed image data buffer 433 
and the raw image data buffer 435 are provided after the 
edge extraction circuit 403, as described with reference to 

25 Figs. 15 and 16, with the N2 data transfer lines 540j and N2 
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data transfer lines 560j outputted therefrom. With this 
construction, the edge extraction circuit 404 generates the 
edge extraction image 1906 and transfers the 
hoise/background-removed image 1904 unaltered. In this case, 
5 similarly in the manner shown in Fig. 16, the switch 20j 
corresponding to each row j is switched to connect the 

M= corresponding data transfer lines 520j, 540j, and 560j 

Q 

O alternately to the corresponding data buffer 171j. By doing 

yQ so, the noise/background-removed image 1904, center-of- 

H= 

JJlO gravity image 1905, and edge extraction image 1906 are 

111 

s combined to generate the composite image 1907. The 

P 

W composite image 1907 is stored in the corresponding data 

ru 

S| buffer 171 j and can be displayed at a desirable timing, 

ftj When it is desirable to perform image processing with 

15 the above system at an even faster frame rate, the analog- 
to-digital converters 210j can perform analog- to-digital 
conversion at a desirable gradation as described in the 
first embodiment. In this case, the comparison image stored 
in the data buffer 1405 should have the same number of bits 
20 as the converted image. Alternatively, it is possible to 
control the data controller 1404 to convert the bit number 
when reading the image from the data buffer 1405. 

As described in the first embodiment, it is also 
possible to combine gradation alteration and binning in the 
25 analog-to-digital conversion to perform image processing 



74 



suitable for the intended target in order to attain an 
adaptive image processing operation. However, in order to 
perform subtraction calculations, matching, or other 
comparison processes on images with different, gradations or 
resolutions in the present embodiment, it is preferable to 
pre-store, in the data buffer, a plurality of comparison 
images (background images or reference images) for each of 
the plurality of possible conditions. 

It is also possible to generate and output only the 
portions, of an image, that are different from a previous 
image. More specifically, it is possible to generate an 
extracted image of a moving object. This is achieved by 
storing the image of a previous frame in the data buffer 
1405, and by subtracting the image of the previous frame 
from the present image, by using the comparison processing 
circuit 1402, to thereby generate an output image. Here, 
instead of performing a simple subtraction with the present 
frame, the image of the present frame can be compared with 
the image of the previous frame (data buffer) , and 
subsequently, the image post-processing circuit 1403 
determines that image signals exceeding a certain threshold 
value are moving pixels. By outputting the thus determined 
image signals as a final output result, it is possible to 
clearly output only those pixels that move in the entire 
image. By calculating the center of gravity based on this 



image output using the center-of-gravity calculating device, 
it is possible to perform precise tracking of moving objects. 

Next, a camera system for high-speed image processing 
according to a third embodiment of the present invention 
will be described with reference to Figs. 21 through 26. 

Fig. 21 shows the construction of the camera system 1 
according to the third embodiment. Parts and components the 
same as or similar to those in the camera system 1 according 
to the first and second embodiments are designated by the 
same reference numerals in the present embodiment to avoid 
duplicating description. 

The image-processing unit 14 of the present embodiment 
is provided with the same image pre-processing circuit 1401 
as described in the second embodiment. The image pre- 
processing circuit 1401 serves as a processing circuit 400 
for performing noise reduction as a pre-process. In 
addition, the image-processing unit 14 is provided with a 
pattern matching circuit 2402. The pattern matching circuit 
2402 serves as a processing circuit 400 for performing a 
comparison process. In addition, the image-processing unit 
14 is provided with a center-of-gravity calculating circuit 
402 as a processing circuit 400 for performing a post- 
process . 

Similar to the second embodiment, a data buffer 2405 
is connected to the pattern matching circuit 2402. The data 
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buffer 2 405 stores ml x m2 reference image data that the 
pattern matching circuit 2 402 uses to perform pattern 
matching. Also as in the second embodiment, the image- 
processing unit 14 of the third embodiment is provided with 
a data controller 2404 for controlling the transfer of the 
ml x m2 reference image data from the data buffer 2 405 to 
the pattern matching circuit 2402. 

As in the first embodiment, the N2 data transfer lines 
250j extending from the output terminals of the analog-to- 
digital converter array 13 form branches, with one branch 
connected to the image-processing unit 14 and the other 
connected to the raw image data buffer 460. As in the first 
embodiment, the output terminal of the raw image data buffer 
460 is connected to the N2 data transfer lines 500j . After 
storing raw image data for a predetermined period of time, 
the raw image data buffer 460 outputs the image data to the 
switches 20j via the data transfer lines 500j . 

As shown in Fig. 22, the image pre-processing circuit 
1401 is configured of N2 bit comparators 1407-j , as in the 
second embodiment. N2 data transfer lines 1440-, are 
connected to the output terminals of the bit comparators 
1407-j. The N2 data transfer lines 1440j are connected to the 
pattern matching circuit 2402. 

Next, the pattern matching circuit 2402 will be 
described with reference to Figs. 22 and 23. 
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The pattern matching circuit 2 402 performs a matching 
operation on a mlxm2 reference image s(p, q) and the raw 
image data D(x, y) , from which noise has been reduced. 

Numerous algorithms for matching operations have been 
proposed. For example, the following function "Error" is 
determined for an ml x m2 region, which has the pixel (x, y) 
in its upper left corner, and the ml x m2 reference image. 
It is possible to determine that these regions match when 
the value of this function is less than or equal to a 
predetermined threshold value. 

ml m2 

ERROR(x, y) = £ £ |z>(* + - 1, y + 3 - 1) - s(p, q)\ 

p=l q=l 

For simplicity, the present embodiment employs a 3 x 3 
reference image as the ml x m2 reference image data. In 
other words, ml = m2 = 3. The 3x3 regions, centered on 
each pixel (x, y) , in the Nl x N2 image are compared to the 
3x3 reference image in a matching operation. Hence, the 
function Error shown below is determined for 3x3 regions, 
each of which is centered on the each pixel (x, y) , and the 
3x3 reference image. When the value of the function is 
equal to or below the predetermined threshold value, the 3x3 
region and 3x3 reference image are said to match. When a 
match is determined, a value of one (1) is outputted for the 
center pixel (x, y) . When the images do not match, a value 
of zero (0) is outputted for the center pixel (x, y) . 
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Through this process, matching image data M(x, y) is 
generated. 

ERROR(x, y) = £ £ |£><x + /? -2,y + < ? -2)- 9 )| 

In a more detailed description, the pattern matching 
5 circuit 2402 includes N2 total sum of differential absolute 
values comparators 2430j which are provided in one-to-one 
correspondence with the N2 data transfer lines 1440j . 

As shown in Fig. 23, each total sum of differential 

J 

absolute values comparator 2 430j is connected to not only 

a 

' y 10 the corresponding data transfer line 1440j , but also the 
J . data transfer line 1440-j-i and data transfer line 1440 j+1 

u . directly above and below the data transfer line 1440j . In 

4 
=• 

f t other words, the total sum of differential absolute values 

y 

comparator 2430j is connected to the corresponding m2 (3 in 
15 the present embodiment) data transfer lines j-1, j, and j+1. 

Within the total sum of differential absolute values 
comparator 2430-j are provided mlxm2 (3x3=9 in the present 
embodiment) differential absolute value processors 2431(0) 
through 431(8), one adder 2432, and one comparator 2433. 
20 Every ml (3 in the present embodiment) differential 

absolute value processors among the differential absolute 
value processors 2431(0) through 2431(8) are connected to 
one of the m2 data transfer lines 1440j_i, 1440j, or 1440j+i. 
In other words, the differential absolute value processors 
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2431(0) through 2431(2) are connected to the data transfer 
line 1440j-i. The differential absolute value processors 
2431(3) through 2431(5) are connected to the data transfer 
line 1440j. The differential absolute value processors 
5 2431(6) through 2431(8) are connected to the data transfer 
line 1440j+i. The differential absolute value processors 
2431 in each row are configured to transfer image data from 
the corresponding data transfer line 1440 to subsequent 
differential absolute value processors in sequence. 
According to this data transfer, image data for a certain 
photodetector element 12 0 and the eight photodetector 
elements surrounding this photodetector element 120 is 
inputted into the input terminals of the differential 
absolute value processors 2431. It is noted that since no 
j-1 row exists for the total sum of differential absolute 
values comparator 2430i corresponding to the first row (j=l) , 
data transfer lines 1440 are not connected to the 
differential absolute value processors 2431(0) through 
2431(2) in the comparator 2430i. Accordingly, the image 
20 data 0 is always inputted into the differential absolute 
value processors 2431(0) through 2431(2) in the comparator 
2430i. Similarly, since no row j+1 exists for the total sum 
of differential absolute values comparator 2430 N2 
corresponding to the N2 th row (j=N2) , data transfer lines 
25 1440 are not connected to the differential absolute value 
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processors 2431(6) through 2431(8) in the comparator 2430 N2 . 
Accordingly, the image data 0 is always inputted into the 
differential absolute value processors 2431(6) through 
2431(8) in the comparator 2430 N2 . 
5 The other input terminal of each of the differential 

absolute value processors 2431(0) through 2431(8) is 
connected to the data buffer 2405. The data buffer 2405 

*■* 

O transfers the mlxm2 (3x3 in this example) reference image 

O 

■W s (p, q) to the other input terminals. 

H10 The output of each differential absolute value 

s 

HI processor 2431 is connected to the corresponding input 

S 

O terminal of the adder 2432. The output of the adder 2432 is 

w 

fy connected to one input terminal of the comparator 2433. A 

O predetermined threshold value transmitted from the 

ru 

15 Instruction/command bus 16 is inputted into the other input 
terminal of the comparator 2433. 

Each differential absolute value processor 2431 
includes a subtractor 2434 and an absolute value processor 
2435. The subtractor 2434 subtracts reference image data 

20 received from the data buffer 2405 from the input image data, 
from which noise has been reduced. The absolute value 
processor 2435 determines the absolute value of the 
difference obtained in the subtraction process. The adder 
2432 determines the total sum of absolute values received 

25 from all the nine differential absolute value processors 
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2431(0) through 2431(8). The comparator 2433 compares the 
total sum of these absolute values with the predetermined 
threshold value. The comparator 2433 outputs a value of one 
(1) when the total sum is less than or equal to the 
threshold value, and outputs a value of zero (0) when the 
total sum is greater than the threshold value. The 
comparison result indicates whether or not the input image 
for an mlxm2 (3x3) region, centered on a pixel (x, y) for 
the noise-reduced image data D(x, y) which is inputted into 
the centered differential absolute value processor 2431(4), 
is equivalent to the mlxm2 (3x3) reference image. An output 
of one (1) indicates that the images are the same, while an 
output of zero (0) indicates they are not the same. 

Each total sum of differential absolute values 
comparator 2430j having the construction described above 
repeatedly performs calculation-and-comparing operation for 
sequential 3x3 regions, each of which is centered on one of 
the Nl pixels (x, y) in the corresponding row j (j=y) . For 
each 3x3 region, the differential absolute values comparator 
2430j calculates absolute values of the differences, and 
compares the calculated results with the threshold value. 
In this way, the differential absolute values comparator 
2 430j sequentially outputs matching image data M(x, y) for 
the corresponding row j . The data transfer line 440j is 
connected to the output terminal of each total sum of 
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differential absolute values comparator 2430j. Accordingly, 
N2 data transfer lines 440j are connected to the output 
terminals of the overall pattern matching circuit 2402. The 
N2 data transfer line 440j are also connected to the center- 
5 of-gravity calculating circuit 402. 

It is noted that the size (mlxm2) of the reference 
image used as the comparison image is not limited to 3x3, 

M- 

O but can be selected arbitrarily according to the size of the 

O 

03 object image to be compared, providing mlxm2 differential 

« 

H=10 absolute value processors 2431 are arranged two- 

Til dimensional ly in ml columns and m2 rows within the total sum 

s 

O of differential absolute values comparator 2430j provided 

y 

fU for each row j . The m2 rows of differential absolute value 

D processors 2431 can be connected to a total of m2 

ru 

15 neighboring data transfer lines 1440, including the 
corresponding data transfer line 1440j. ml differential 
absolute value processors 2431 are provided in each row that 
corresponds to one data transfer line 1440. 

As in the first embodiment, the center-of-gravity 

20 calculating circuit 402 has the construction shown in Fig. 7. 
The center-of-gravity detecting circuit 409 performs a 
center-of-gravity operation on the matching image data M(x, 
y) inputted from the pattern matching circuit 2402 via the 
N2 data transfer lines 440j to find the center-of-gravity 

25 position (Px, Py) . This result is outputted to the control 
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circuit 15. The center- of-gravity image data generating 
circuit 425 in the center-of-gravity calculating circuit 403 
generates the center-of-gravity image data P(x, y) 
indicating the center-of-gravity position (Px, Py) and 
outputs this data to the data transfer lines 520j. 

As shown in Figs. 21 and 24, the switch 2 0j 
corresponding to each row j switches between the data 
transfer line 500j from the raw image data buffer 460 and 
the data transfer line 52 0j from the center-of-gravity 
calculating circuit 402 to connect the data transfer lines 
500j and data transfer lines 520j to the corresponding data 
buffer 171 j. Therefore, the raw image data and center-of- 
gravity image data can be combined and stored in the data 
buffer 171-j. 

The camera system 1 of the present embodiment having 
the construction described above can monitor movement of an 
object at a fast frame rate. The camera system 1 can 
extract an important moment, such as the moment in which the 
object's moving state changes, and display n frames before 
and n frames after that moment on the monitor. 

Next, the above-described processes in the present 
embodiment performed under the control of the control 
circuit 15 will be described according to the flowchart in 
Fig. 25. 

Beginning in S100, the control circuit 15 controls the 
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photodetector array 11 and analog- to-digital converter array 
13 to acquire one frame of raw image data. The raw image 
data is transferred to the image-processing unit 14 via the 
data transfer lines 250j and stored in the raw image data 
5 buffer 460. 

Next, the control circuit 15 directs the image- 
processing unit 14 to perform image processes in S110. 

M> 

O First the image-processing unit 14 directs the image pre- 

w 

03 processing circuit 1401 to perform noise reduction on the 

H=10 image data. The image-processing unit 14 then controls the 
fU pattern matching circuit 2402 to perform a matching 

O operation on the noise-reduced data and the mlxm2 reference 

W 

fU image showing the targeted object. Through these processes 

%j 

0 the image-processing unit 14 generates matching image data 

1 y 

15 M(x, y) . The position of the object in the present frame is 
indicated by the position of a data (1) in the matching 
image data. 

In S120, the control circuit 15 directs the center-of- 
gravity calculating circuit 402 to calculate the center-of- 

2 0 gravity position (Px, Py) in the matching image data M(x, y) 
for the current frame. The result of this calculation 
indicates the center-of-gravity position (Px, Py) of the 
object in the current frame. This result is transmitted to 
the control circuit 15. The center-of-gravity image data 

2 5 generating circuit 42 5 in the center-of-gravity calculating 
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circuit 402 generates center-of-gravity image data. 

In S130, the control circuit 15 calculates the amount, 
by which the speed of the object presently changes, based on 
the center-of-gravity position (Px, Py) of the object 
obtained in S120. More specifically, the control circuit 15 
includes a memory (not shown in the drawings) for storing 
speed data and center-of-gravity position data for the 
previous frame. Hence, the control circuit 15 determines 
the current speed of the object by calculating the 
difference between the center-of-gravity position (Px, Py) 
of the object from the current frame and that from the 
previous frame. Next, the control circuit 15 determines the 
speed-changing amount by calculating the absolute value of 
-the difference between the current speed data obtained for 
the current frame and the speed data obtained for the 
previous frame. 

In S140, the control circuit 15 compares the current 
speed-changing amount, calculated in S130, to a 
predetermined set value. When the current speed-changing 
amount is less than or egual to the set value (No in S140) , 
composite image data for the current frame is generated and 
stored in the data buffers 171 in S150. In other words, by 
controlling the switches 20j, raw image data from the raw 
image data buffer 460 and center-of-gravity image data from 
the center-of-gravity image data generating circuit 425 are 
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combined and stored in the data buffers 171. The process 
then continues by obtaining and processing the next frame 
(S100) . 

On the other hand, if the current speed changing 
amount is larger than the set value (Yes in S140) , it is 
determined that the moving state of the object has changed. 
Accordingly, in S160 the control circuit 15 controls the 
signal conversion controller 19 to start a monitor display 
process. After directing the signal conversion controller 
19 to begin this process, the control circuit 15 begins 
acquiring and processing the next frame in S100. 

After receiving the command to perform a monitor 
display process, the signal conversion controller 19 begins 
directing the signal converter 17 in S170 to sequentially 
display composite image data for n frames already stored in 
the data buffers 171, composite image data for the current 
frame, and composite image data for n subsequent frames. 
Hence, this process is the same as step S70 in the first 
embodiment (Fig. 12) . 

By performing the operations described above, the 
camera system 1 of the present embodiment can continuously 
take images showing the state of a swinging bat, for example, 
at a high frame rate, as shown in Fig. 26. The camera 
system 1 can extract and display the frame showing the 
instant at which the ball, which is the moving object, meets 
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the bat and several frames before and after this instant. 

More specifically, the photodetector array 11 and 
analog-to-digital converter array 13 acquire images for a 
plurality of consecutive frames at a high frame rate (S100) . 
In the image processing unit 14, the pattern matching 
circuit 2402 performs a pattern matching process on the 
acquired image data (S110) . In this step, an image of the 
ball, which is the target object, is used as the mlxm2 
reference image, thereby enabling the ball to be extracted 
from the image data. Next, the center of gravity is 
calculated for the extracted ball (S120) . Position data for 
the center of gravity of the ball is transferred to the 
control circuit 15, which then calculates the velocity of 
the ball based on the difference from the position in the 
previous frame (S130) . When the ball meets the bat, the 
velocity and direction of the ball change abruptly. It is 
possible to extract the frame at that moment (S140) . The 
frame for this moment and the frames before and after this 
moment are set as the final output <S160 and 2170) . 

According to this process, images for a plurality of 
frames (T = 1 - 6) showing the bat swinging are acquired 
consecutively at a high frame rate. The moment at which the 
ball hits the bat (T = 4) is detected. Only the image of 
the moment at which the ball meets the bat (T = 4) and the 
previous and following images (T = 3, 5) are extracted. It 



is then possible to overlay the position of the ball's 
center of gravity in the display of these images. 

In this way, it is possible to extract images 
appropriate to the observation objective from an enormous 
5 amount of data acquired at a high frame rate. 

If the image acquisition rate of the photodetector 
array 11 and the analog-to-digital converter array 13 
according to the present embodiment is 1,000 frames/sec, for 
03 example, images can be taken for every four centimeters the 

If 10 ball moves when the ball is traveling at 150 km/hr. 
PJ Moreover, since the image-processing unit 14 performs image 

C-3 processes in real-time, only desired frames at which the 

W 

PJ ball meets the bat can be extracted. Accordingly, the 

O images can be displayed on the monitor 18 at a normal frame 

fU 

15 rate (such as 30 frames/sec) and inputted into a general- 
purpose image processor supporting NTSC camera input (about 
30 frames/sec) . Further, since the result of the center-of- 
gravity process is displayed over the raw image, it is 
possible to reduce the amount of load placed on subsequent 

2 0 image-processing devices. 

Accordingly, the camera system 1 of the present 
embodiment can monitor a moving object at a high frame rate, 
extract images only for the desired moment of observation, 
and output images just before and after that moment. For 

25 example, when inspecting a sheet metal drilling process in 
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real-time, it is possible to extract the image at the moment 
the drill bit, which is the object of observation, contacts 
and drills through the sheet metal. It is possible to 
display this image and images before and after this image on 
the monitor. It is therefore possible to measure the 
rotational speed and moving speed of the drill bit, bending 
state of the plate, and the like for this moment. These 
measurements can be used for conducting more precise 
inspections and uncovering the causes of abnormalities. 

The control process described above calculates the 
velocity of an object and extracts images when the speed 
variation of the object exceeds a predetermined value. 
However, the camera system 1 of the present embodiment can 
also determine the position of the object, as in the first 
embodiment. For example, the camera system 1 can determine 
when the object, the drill bit for example, reaches the 
sheet metal . 

In the present embodiment, the pattern matching 
circuit 2402 includes N2 total sum of differential absolute 
values comparators 2430 which are connected to the N2 data 
transfer lines 1440. However, the pattern matching circuit 
2402 can also be configured of a smaller number of total sum 
of differential absolute values comparators 2430, that is, M 
total sum of differential absolute values comparators 2430, 
where M < N2 . In this case, switches are provided at the 
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input terminals of the M total sum of differential absolute 
values comparators 2430. The control circuit 15 estimates a 
position at which a match of the object can be found in the 
current frame based on the center-of-gravity position (Px, 
Py) detected in the previous frame. The control circuit 15 
connects the M total sum of differential absolute values 
comparators 2430 to M neighboring data transfer lines 1440j 
through 1440 j+M -i that correspond to a region including the 
estimated position. The pattern matching circuit 2402 
outputs matching image data M(x, y) for these M rows j 
through j+M-1 and outputs the data 0 for all other data 
transfer lines. As a result, matching image data M(x, y) 
corresponding to all NlxN2 pixels is generated and inputted 
into the center-of-gravity calculating circuit 402 . With 
this configuration, the pattern matching circuit 2 402 can 
generate matching image data using a more simplified 
construction. 

Next, a camera system for high-speed image processing 
according to a fourth embodiment of the present invention 
will be described with reference to Figs. 27(a) and 27(b). 

In the present embodiment, an extraction timing is 
determined by comparing the acquired images to external data 

Fig. 27(a) shows the construction of the camera system 
1 according to the fourth embodiment of the present 
invention. The camera system 1 is similar to the camera 



system 1 of the second embodiment, having an image pre- 
processing circuit 1401 for performing noise reduction. 
However, a pattern matching circuit 3402 is provided after 
the image pre-processing circuit 1401. 

As in the second embodiment, the image pre-processing 
circuit 1401 includes N2 bit comparators 1407j. N2 data 
transfer lines 1440j are connected to the output terminals 
of the bit comparators 1407j and also to the pattern 
matching circuit 3402. 

In the present embodiment, the pattern matching 
circuit 3402 performs a matching operation with an mlxm2 
reference image on the raw image that has passed through the 
noise reduction pre-process. In this example, ml and m2 are 
set to 3, as in the third embodiment. The pattern matching 
circuit 3402 differs from the pattern matching circuit 2402 
in the third embodiment in that the pattern matching circuit 
3402 is configured of only one total sum of differential 
absolute values comparator 2430 2 . The total sum of 
differential absolute values comparator 2430 2 is connected 
to a single corresponding line from among the N2 data 
transfer line 1440j (in this example, the data transfer line 
1440 2 ) and the lines directly above and below this line (in 
this example, the data transfer lines 1440 x and 1440 3 ) . 

The total sum of differential absolute values 
comparator 2430 2 has the same configuration as that shown in 
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Fig. 23. However, the output terminal of the comparator 
2433 is connected to the control circuit 15 via the 
instruction/command bus 16. 

The pattern matching circuit 3402 allows the N2 data 
transfer lines 1440j from the image pre-processing circuit 
1401 to be fixedly connected directly to the corresponding 
data buffers 171j . Accordingly, after undergoing the noise 
reduction pre-process in the image pre-processing circuit 
1401, the raw image data in the present embodiment is stored 
directly in the data buffers 171j without being combined 
with other image data. Therefore, the switches 2 0j are not 
provided in the present embodiment . 

According to the present embodiment, a search image 
s(x, y) (having a size of ml x m2) representing the object 
is pre-stored in the data buffer 2405. When the input image 
D(x, y) for a certain frame is transferred to the image- 
processing unit 14, the mlxm2 search image s(p, q) stored in 
the data buffer is retrieved to perform the matching 
operation . 

The pattern matching circuit 3402 calculates whether 
any of the Nl number of mlxm2 regions, each of which is 
centered on one of the Nl pixels of the second row in the 
photodetector array 11, matches the mlxm2 reference image. 
That is, the total sum of differential absolute values 
comparator 2430 2 calculates the following difference 
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equation. 



3 3 

ERROR{x,2) = £ £ \ D ^ X + P-22 + 9-Z)- s(P> l)\ 



5 The total sum of differential absolute values 

comparator 2430 2 outputs a value of one (1) when the result 

O of calculating the function Error is less than or equal to a 

predetermined threshold value, and outputs a value of zero 

*Z (0 when the result is greater than the threshold value. 

rUlO Hence, the function Error is obtained for each of the 

a 

■ : Nl mlxm2 regions centered on each of the Nl pixels in the 

w 

fU second row (x, 2) for an image of a certain frame. If some 

Si 

O of the results calculated for the function Error is less 

re 

than or equal to the predetermined threshold value, that 

15 frame is set as the desired timing. 

Accordingly, the pattern matching circuit 3402 detects 
the degree of conformity between the input image D(x, y) and 
the comparison image s(p, q) in real-time. When an error 
less than or equal to the threshold value is obtained, the 

2 0 image data for the frame at that time and the image data for 
the previous and subsequent frames are set as output images. 
The signal conversion controller 19 directs the signal 
converter 17 to convert these output images to image signals. 
The control circuit 15 in the present embodiment 
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performs the process shown in the flowchart of Fig. 27(b) . 
Specifically, as in S10 of the first embodiment (Fig. 12) , 
the control circuit 15 obtains the image for each frame in 
S210. In S220, the image pre-processing circuit 1401 
5 executes a noise reduction process on the acquired image, 
and the pattern matching circuit 3 402 executes the matching 
H process on the image. In S230, the control circuit 15 

O determines if any of the Nl comparison results sent 

sequentially from the total sum of differential absolute 
%0l0 values comparator 2430 2 is equal to one (1) . If none of the 

ru 

3 Nl comparison results is equal to one (1) (No in S240) , the 

o 

W raw image for the current frame is stored in S250 and the 

Nl process returns to S210 to process the next frame. On the 

fU other hand, if at least one of the Nl comparison results is 

15 equal to one (1) (Yes in S240) , the current timing is 
determined to be the extraction timing and the process 
shifts to S260. In S260, a monitor display instruction is 
outputted to the signal conversion controller 19. In S270, 
a display/outputting operation is attained. Accordingly, 
2 0 noise-reduced raw image data outputted from the image pre- 
processing circuit 1401 for the current frame and the 
previous and subsequent frames is displayed on the monitor 
18. With this process, video images before and after a 
desired timing can be displayed on the monitor 18. 
25 For example, it is possible to extract images at the 
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moment the chip 53 is mounted on a substrate, as described 
with reference to Figs. 14(a) through 14(c) . In other words, 
the matching operation determines whether any of 
predetermined regions, in an input image, that represent 
predetermined positions on a substrate (corresponding to 
positions in the second row j of the photodetector array 11 
in the present embodiment) matches an image showing the chip 
53 as the target image. In doing so, it is possible to 
extract the moment at which the chip 53 is mounted at any of 
the predetermined positions on the substrate and to output 
images at this timing. 

A particular aspect of the present apparatus is that 
it transfers signals in parallel from the plurality of rows 
in the photodetector array 11. Moreover, the image- 
processing unit 14 achieves high-speed image processing by 
performing parallel processes. More specifically, the image 
pre-processing circuit 1401 includes the plurality of bit 
comparator 1407j, one for each row, in order to perform 
noise reduction on all the rows in parallel. The pattern 
matching circuit 3402 includes ml differential absolute 
value processors 2 431 for each of corresponding m2 rows (the 
first, second, and third rows in the example described 
above) and performs a matching operation by determining the 
absolute values of the differentials for the respective rows 
in parallel. When tracking an object moving at a high speed, 
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the shape of the object's image varies when the rotation or 
orientation of the object changes. Therefore, it is 
difficult to display the object in real-time at a slow frame 
rate (30 Hz) , such as that of a conventional TV camera, 
because an extremely complex algorithm is necessary to find 
the same object in images of different frames. However, 
when capturing images at a high frame rate (> 1 KHz) , as 
with the apparatus of the present embodiment, there is only 
a little difference in images between frames. Therefore, it 
is possible to track the object accurately by performing a 
matching calculation at a high rate of speed using the 
simple algorithm described above. 

The camera system for high-speed image processing 
according to the present invention is not limited to the 
embodiments described above. Many modifications may be made 
therein . 

In the embodiments described above, the image 
processing device 14 for parallel processing is provided 
with one processing element for each row of an image in 
order to perform image processing in parallel. By 
performing image processing in parallel, it is possible to 
operate at a high speed necessary for such large-scale 
calculations as image data. 

In addition to the examples described above, another 
method of parallel processing employs a block parallel 



structure (see Fig. 28) . Here, the photodetector array is 
divided into a plurality of blocks. A plurality of 
processing circuits 400a (each including the binarization 
circuits 401, center-of-gravity calculating circuit 402, and 
5 edge extraction circuit 403) are provided with a one-on-one 
correspondence to the plurality of blocks. It is also 
possible to use a complete parallel structure (see Fig. 29) 
Q for performing a completely parallel process. In this 

iQ method, a plurality of processing circuits 400b are provided 

y=10 with a one-on-one correspondence to the photodetector 
ill elements 120 arranged two-dimensionally in the photodetector 

Q array 11. The method of parallel processing can be selected 

fy according to the desired application, desired level of 

O integration, or required processing speed. In the complete 

fu 

15 parallel processing method (Fig. 29) , the processing 
elements disclosed in Japanese unexamined patent application 
publication no. HEI-10-145680 , for example, can be used as 
the processing circuits 400b. The processing circuits 400b 
can achieve a completely parallel process by performing a 

2 0 total sum calculation, center-of-gravity calculation, or 
other calculation based on image signals received from the 
corresponding photodetectors . Accordingly, the apparatus 
can determine whether the result of the center-of-gravity 
calculation or the total sum calculation is equivalent to a 

2 5 predetermined set value, for example. The apparatus can 
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determine that the extraction timing has reached when the 
calculation result becomes equivalent to the predetermined 
set value, and display the images of a frame for this timing 
and for previous and subsequent frames. A feature of the 
5 block parallel method is its efficiency in finding a match 
or a center of gravity for each region, while the complete 
parallel method achieves speed enhancement. Both methods 

M= 

O employ an architecture, in which an analog-to-digital 

D 

Cy converter 210 is provided for each line or each block to 

m 

jh&lO perform efficient analog-to-digital conversion and the 
fjj processing circuits 400a or 400b are arranged in the 

□ partially parallel manner or in the complete parallel manner 

W 

Hi to perform high-speed image processing. 

O In the first through third embodiments described above, 

ru 

15 a plurality of image signals are combined together and 
outputted. That is, in the first embodiment, a raw image 
signal, center-of-gravity image signal, and edge extraction 
image signal are combined and outputted. In the second 
embodiment, the center-of-gravity image signal, edge 

20 extraction image signal, and possibly also the raw image 
signal that has undergone noise reduction and background 
removal are combined and outputted. In the third embodiment, 
the raw image signal and center-of-gravity image signal are 
combined and outputted. However, it is not necessary to 

25 combine these signals. That is, a raw image signal 
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generated by the analog-to-digital converter array 13, or 
either one of the processed image signals generated by the 
processing circuits 400 in the image-processing unit 14 can 
be outputted unaltered. In this case, the switches 20j are 
unnecessary. Hence, similarly as described in the fourth 
embodiment, each data buffer 171j can be fixedly connected 
to some data transfer line that is used to transfer image 
signals desired to be outputted. 

In the above-described embodiments, the plurality of 
analog-to-digital converters 210 are provided in the analog- 
to-digital converter array 13 in one-to-one correspondence 
with the plurality of rows j in the photodetector array 11. 
However, it is also possible to provide the analog-to- 
digital converters 210 in one-to-one correspondence with the 
plurality of columns i, as shown in Fig. 30. In this case, 
the analog-to-digital converter array 13 includes Nl analog- 
to-digital converters 210. The N2 photodetector elements 
120 in each column i are electrically connected together in 
the photodetector array 11 and connected to the 
corresponding analog-to-digital converter 210. In each 
column i, by switching the vertical scanning signal 
seguentially , the photoelectric output from each of the N2 
photodetector elements 120 can be read sequentially. In 
this case, the processing circuits 400 in the image- 
processing unit 14 can perform parallel processing by 
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processing output signals from the plurality of columns i. 

In the above-described embodiments, the analog-to- 
digital converter 210 includes the charge amp 221. However, 
the analog-to-digital converter 210 and the charge amp 221 
can be provided separately as shown in Fig. 31. As shown in 
this figure, an amp array 12 is connected to the 
photodetector array 11. The amp array 12 has N2 charge amps 
221. The analog-to-digital converter array 13 is provided 
between the amp array 12 and the parallel processing unit 14 
The analog-to-digital converter array 13 has N2 analog-to- 
digital converters 210. With this construction, each amp 
22 0 in the amp array 12 successively converts charges, 
outputted from the Nl photodetectors 12 0 on the 
corresponding row 110 of the photodetector array 11, into 
voltage signals. These analog voltage signals are outputted 
to the corresponding analog-to-digital converter 210 in the 
analog-to-digital converter array 13. The analog-to-digital 
converter 210 successively converts the analog voltage 
signals to digital signals, and supplies the digital signals 
to the parallel processing unit 14. 

While a single photodetector array 11 is provided in 
the embodiments described above, it is also possible to 
provide a plurality of photodetector arrays 11. 
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INDUSTRIAL APPLICABILITY 
The camera system for high-speed image processing 
according to the present invention has a wide variety of 
applications, including a real-time scanner in automated 
machine tools and monitoring systems for observing various 
obj ects . 
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CLAIMS 

1. A camera for high-speed image processing, 

comprising: 

a photodetector array having a plurality of 
photodetectors , which are arranged two-dimensionally in a 
plurality of rows and in a plurality of columns and which 
are divided into a plurality of blocks, the photodetector 
array repeatedly receiving optical images and generating 
output signals for the optical images in a plurality of 
consecutive frames at a predetermined frame rate; 

an analog-to-digital converter array including a 
plurality of analog-to-digital converters in one-on-one 
correspondence with the plurality of blocks in the 
photodetector array, each analog-to-digital converter 
performing analog-to-digital conversion of the output 
signals which are read sequentially from the photodetectors 
in the corresponding block; 

an image-processing unit performing predetermined 
parallel processes on digital signals, which are transferred 
from the analog-to-digital converter array and which 
correspond to the signals outputted from the photodetectors, 
thereby generating a processed result signal indicating the 
results of the processes; 

a selector selecting at least one frame based on the 
processed results obtained by the image-processing unit; 
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a signal converter converting, into an image signal of 
a desired frame rate, at least one of the processed result 
signal and the output signal from the analog-to-digital 
converter array, the signal converter outputting the image 
5 signal; and 

a signal conversion controller controlling the signal 
converter to perform the image signal conversion operation 

U 

D for the at least one frame selected by the selector. 

Q 

00 2. A camera for high-speed image processing as 

M=10 claimed in claim 1, wherein the desired frame rate is lower 
ry than the predetermined frame rate. 

s 

Q 3. A camera for high-speed image processing as 

yj 

fU claimed in claim 1, wherein the signal converter converts 

%j 

d either one of the output signal from the analog-to-digital 

ru 

15 converter array and the processed result signal, into the 

image signal of the desired frame rate, and outputs the 
image signal. 

4. A camera for high-speed image processing as 
claimed in claim 1, wherein the signal converter combines 

20 the output signal from the analog-to-digital converter array 
and the processed result signal, converts the combined 
signal into the image signal of the desired frame rate, and 
outputs the image signal. 

5. A camera for high-speed image processing as 
25 claimed in claim 1, wherein the signal converter has a 
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buffer memory at a signal input side thereof, the buffer 
memory storing at least one of the output signal from the 
analog-to-digital converter array for at least several 
frames and the processed result signal for the at least 
5 several frames. 

6. A camera for high-speed image processing as 
claimed in claim 1, further comprising a data buffer storing 
p predetermined process data, the image-processing unit 

rg performing a predetermined parallel process, using the 

1^10 predetermined process data, onto the digital signals that 
ry are transferred from the analog-to-digital converter array 

p and that correspond to the signals outputted from the 

w 

ry photodetectors . 

Q 7 . A camera for high-speed image processing as 

ru 

15 claimed in claim 1, wherein the plurality of analog-to- 
digital converters in the analog-to-digital converter array 
are provided in one-on-one correspondence with the plurality 
of rows of photodetectors in the photodetector array. 

8 . A camera for high-speed image processing as 
20 claimed in claim 1, wherein the plurality of analog-to- 
digital converters in the analog-to-digital converter array 
are provided in one-on-one correspondence with the plurality 
of columns of photodetectors in the photodetector array. 

9. A camera for high-speed image processing as 
25 claimed in claim 1, wherein the image-processing unit 
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includes a plurality of processors in one-to-one 
correspondence with the plurality of photodetectors , the 
plurality of processors performing parallel processes on the 
digital signals that are transferred from the analog-to- 
digital converter array and that correspond to the signals 
outputted from the plurality of photodetectors. 

10. A camera for high-speed image processing as 
claimed in claim 1, wherein the image-processing unit 
includes at least one parallel processing circuit, each of 
the at least one parallel processing circuit performing a 
corresponding parallel process on the digital signals that 
are transferred from the analog-to-digital converter array 
and that correspond to the signals outputted from the 
photodetectors, thereby outputting a processed result signal 
indicative of the processed result, the selector selecting 
at least one frame based on at least one process result from 
the at least one parallel processing circuit, the signal 
converter converting at least one of the output signal from 
the analog-to-digital converter array and the processed 
result signal obtained by the at least one parallel 
processing circuit, into the image signal of the desired 
frame rate, and outputting the image signal. 

11. A camera for high-speed image processing as 
claimed in claim 10, wherein each of the at least one 
parallel processing circuit is provided with several 
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processing elements in one-to-one correspondence with 
several blocks that make up at least a portion of all the 
plurality of blocks in the photodetector array, each of the 
several processing elements performing a predetermined 
5 parallel process on the digital signals that are transferred 
from the corresponding analog-to-digital converter and that 
are equivalent to the signals outputted from the 
j^f photodetectors existing in the corresponding block. 

* 12. A camera for high-speed image processing as 

? *10 claimed in claim 10, wherein the plurality of analog-to- 
fy digital converters in the analog-to-digital converter array 

H are provided in one-to-one correspondence with the plurality 

y=j 

lu 0 f r ows of photodetectors in the photodetector array, each 

of the at least one parallel processing circuits including 

m 

15 several processing elements which are provided in one-to-one 
correspondence with several rows that make up at least a 
portion of all the plurality of rows in the photodetector 
array, each of the several processing elements performing a 
predetermined parallel process on digital signals that are 

2 0 transferred from the analog-to-digital converter array and 
that are equivalent to the signals outputted from the 
photodetectors in the corresponding row. 
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ABSTRACT 

The analog-to-digital converter array 13 includes one 
analog-to-digital converter 210 for each row of 
photodetectors 120 in the photodetector array 11. The 
image-processing unit 14 includes the plurality of 
processing circuits 400 for performing high-speed image 
processing. The signal converter 17 combines the output 
signals from the analog-to-digital converter array 13 with 
output signals from the image-processing unit 14. Under 
control of the control circuit 15 and the signal conversion 
controller 19, the signal converter 17 downconverts the 
composite signal at an important timing to a frame rate 
suitable for display on the monitor 18 and subsequently 
displays the signal on the monitor 18. 



108 



10/ 0891 



1/32 



CD 



oozz 

OO 
OQ 



cJ3±cJ3ANOO 1VNDIS 



T 



A-A 
W 



unodio cos 

ONISS300Ud°§ 



unodio 

ONISS300Ud 



unoaio 5 § 

ONISS300cJd °3 



o 

00 
CO 
LU 

C|_ 

o— 

Q_— 5 

I 

LU 

O 
< 



CO 



— CD 




2/32 



FIG.2 



13 14 17 




4/32 



FIG.4 



220 

/ 



HI 



rlf 



C11 



Cl" SW11 



31 



C12 



C2 M SW12 



C13 



C3 M SW13 



C14 



C4" SW14 




221 
223 



C21 


-A- 

SW21 


C22 


5W22 


J- 
C23 


-A- 

SW23 


Jt 
C24 


-A- 

SW24 







R 



on / ^ o Q 



5X32 
oo 




00 





cr 






o 






h- 












o 









d3XIW 1VN0IS 
NOIlVZINOdHONAS 



d3±y3ANOO 

oonvNv-oi-nviiDia 



7 



LJJ X 

< 5:0-0' 



tr 

LU 

LU 
> 

O 

o 



a 

CO 



unodio 

ONISS300i: 


CO 

Id ° 








unoHio cm 

0NISS300dd 5 








nr 

DNIS' 


lOdIO - 
33QOdd ° 



a 

CO 
CO 
LU 

St 
£§ 

I 

LU 
O 
< 



f 

CD 
LO 



LU 
<< 



CD 
^— CD 
/ LO 



Kg 
LO 




10/ 089192 



6/32 



o 



o o 

<N CM 



o 

CM 



O 
CM 



O 
CM 



" o o o 

: V »- >- 



LO 
CM 

Tj- CD 

T CM 

J lo 



r\ r 

CD 

CM 

LO 



o 

CM 

o 
h- 



iinOcJIO 0NI1VU3N30 

VIVO 30VIAII 
AlIAVcJD-dO-U31N30 



1 



CD 

LO~" 



^ > 

1 



UflOHIO ONI103130 
AJJAVHO-JO-H31N30 



"3 5 3jT~ 







CD 


CD 






5^ 


LO 



cJ3ddng viva bdviaii asssdooad 



CO 
CO 



CD 
CO- 
LO 



A-N 



CO 



CD 
CS,\ 




— C-J CO 

OOO 
CM CM CM 

OOO 
1 — I — I — 



CD 

8\| 



60 ^ 



CD 

LO CD 



— CD 



CO 

d 



O 



c-j 
z: 

X 



20/ 0891 92 



7/32 




20/ 08 



1 



8/32 



CD- A 




10/ u 



9/32 



oo 





cc 






o 


















o 









uaxiw nvNDis 

N0LLVZIN0UH0NAS 



d31d3ANOO 
DOlVNV-01-lVlIOig 



or 

LU 

LU 
> 




CT>hD 
lu<Oo 



o a: 

I — — ij-Ll 

_i£2>cr 

<Q2< 
■Z. I O 
< O 




11/32 



ui 



O 

o 

x 

is 

< 

Q 

_! 
LJJ 

X 



q b > 

<LU 

I" 



3 + 
_J x 

f 



< 

a 
_j 

LU 

X 
CL 



ca 
4 



asw 



991 




asw 



as~i 



o 

Q 



cr 



2X 




o 
o 



< 

Q 
_l 

LU 

X 
0- 

h> 

QQ X 
4-Q 



Z > 

=> + 
_J^>< 
Oq. 

o 

r 

I 

Q 
_J 
LU 
X 
CL 
I- 

S 
4 



8SIAI 



CM 



LU 



Q 

LU 
CO 

CO CO 



SI 

CL II 

<r 



O > 
CLQ 



=>CL 

GO 

LU 

cr 



Q Q 
LU 

CM CO 
Q 00 
LU 

5 8 

§ CL 



o 

Ll_ 



o 



O 
•— i 

ll. 



CD 
Ll 



12/32 



10/ 089192 



FIG. 12 



c 



START 



ACQUIRE 
IMAGE 



PROCESS 
IMAGE 



-S10 



-S20 



COMPARE 
CENTER-OF-GRAVITY 
RESULT WITH SET POSITION 



-S30 



SI 



S40 



DOES 

CENTER-OF-GRAVITY^ 
POSITION = SET 
^POSITION 

[NO 



YES 



STORE 
IMAGE 



-S50 



> 


r 




SEND DISPLAY 
COMMAND 


> 






DISPLAY/ 
OUTPUT 




r 



-S60 



-S70 



c 



RETURN 



20/ 0891 92 



13/32 



FIG. 13(a) 



IMAGE 
TRANSFER 



IMAGE 
PROCESSING 



IMAGE 
DISPLAY 




-> TIME 



-> TIME 



> TIME 



1 FRAME 



FIG.13(b) 



IMAGE 
TRANSFER 



IMAGE 
PROCESSING 



IMAGE 
DISPLAY 




->TIME 



>TIME 



>TIME 



1 FRAME 



SO/ 03929 

14/32 




FIG. 14(c) *K 



52 



53 





50 



SO/ GQ9192 



15/32 



o 

CNJ 
O 

H 



o 

CM 



i— CM CO 

O O O 

CM CNJ CM 



OJ 

CD 



o o 



LO 
CVJ 



o 

I— 
A 



O 



O 
h- 



d3ddna viva 

3OVM0iSS3OOdd 



JLinOUIO ONUVU3N30 

VIVO 3DV1AII 
AlIAVttO-dO-U31N30 



1 



linOHIO 0NI103130 
A±IAVcJQ-JO-U31N30 
~T * 



W 



O 







LO 


to~v 

LO 



aaddna viva aovra mvh 



J 



1 — ' 

/ i— CM CO 

CM CM CM 
OOO 



LI. J \ 



-TV 

YV 



-TV 



CD 
LO 



LiJ 



<!=!r 
CD 



OJ LO 



LO 

a 

LL. 



go 0 



CM 

X 



16/32 



ill 



w 
M 



00 





cr 






O 
H; 












O 




v — ' ^ 



CD 




uaxiw ivnois 

NOUVZINOdHONAS 



CO 



d31d3ANOO 
DOnVNV-Ol-lVlIDIQ 



JL 



1 7 



1 



cr 

LU 
H 

cr 

LU 

> 

o 
o 




cr>i-z) 

LU<OQ 

•-cr^cr 



17/32 



CD 



in c£> 



U3T10U1N00 
N0ISU3AN00 1VN0IS 



Ot 

ZD 
HO 



O 



o 



M 
W 



IP 

00 O 





cr 






o 






h- 












O 









.CO 



ti31U3ANOO 1VN0IS 



LO 

CD 



ysddna viva 



d3niodiNoo viva 



unouio 

0NISS300dd-lS0d 30VWI 



unouio 

DNIcJVdJAlOO 3DVIAII 



CVl 



unouio 

ONISS300tid-3tid 30VI/MI 



o 
-z. 

00 
00 
ULI 

o~ 

Q_- 

I 

UJ 

a 
< 




10/ 0891 



48/32 



t- CM CO 

o o o 

CM CM CM 

o o o 

1 — I — I — 



CD 
CM 



Oil 



j 


1 i 




r 






cE> 


CD 




OJ 


OJ 


OJ 


LO 


LO 


LO 



LO 
CNJ 



linoaiO DNUVU3N30 

VIVO 3DVWI 
AlIAVcJ0-dO-cJ31N30 



<5 


C3 


tS 


I CO 
CO 


LO ^ 


LO v 


LO 




aaddna viva bdviaii a3ss3ooud 


LO V 


CD 
CO-. 
LO X 


CO 

CO-> 
LO x 

























CO 




CO 




CO 














10/089192 



19/32 
oo 

r— 

d. 

\ CC 

o 



'—I , 




O 

LL. 



H3XIKI 1VN0IS 
NOLLVZINOUHONAS 



oo 



U31U3ANOO 

oonvNV-oi--iviioia 



— ^ 





or 

LU 

i— 
cr 

LU 
> 

-z. 
o 
o 



=1 < 
-z. 
a 

GO 



CM 



0>- Si— 




10/ 089192 



20/32 




*? 0/ 08919 



21/32 



CD 



U3TIOH1NOO 
NOISdBANOO 1VNOIS 



ob 

On 



°9 

00 O 





cr 






o 


















O 









.CO 



U31ti3ANOO 1VNDIS 



id3ddna viva 



H3-TIOU1NOO VIVQ 



o 

CM 



Csl 

d 




unodio DNiivinonvo 

AlIAVaD-dO-a31N30 



unouio 

0NIH01VN Nd311Vd 



imouio 

DNISS300Ud-3dd 3DVIAII 



eg 

o 

00 
00 
LU 

& 

I 

LU 

a 
< 



CD 

CD-s 
LO \ 



CD 
/ LO 



CD 
-CD 
LO 



LU 

<<LU 
~<U- 

cr 



V CO 



"^6 6 6 



H- 1 



04 

X 



22/32 



CM CO 

o o o 

CM CM CM 

o o o 

I— I— h- 



CM 
CM 

O 

LL 



C"4| 









I 3 


r 


r 


r 




CD 


cS 


<3 




OJ 








LO 


LO 


LO 





linOdlO DNIiVcJ3N3D 

viva 30VIAII 
A1IAVUCHO-H31N30 



CD 

cr> 



iinocdio ONiio3i3a 

AlIAVdO- dO-cJ31N30 

* — 



>— CM CO 

ooo 

a CMCMCM 

cvj OOO 
I — I — I — 




10/ 089192 



24/32 



O 
El 





U3XII/M 1VN9IS 
NOIlVZINOdHONAS 



CO 



oonvNV-oi-nviioia 



— ^ — ^ 



-.00 



1 



cr 

LU 

> 

o 
o 



= < 



= <CLX0 



00 







Y 















c 




iljO 
O 



<<<L1_ 
— CO 



tr>H-z> 
lu<oo 

goto 

O 1 Q 



< o 



10/089192 



25/32 



FIG.25 



START 



ACQUIRE 
IMAGE 



-S100 



PERFORM MATCHING 

PROCESS 
(EXTRACT OBJECT) 



CALCULATE CENTER OF 

GRAVITY OF OBJECT 
(POSITION OF OBJECT) 



CALCULATE 
SPEED-CHANGING 
AMOUNT OF OBJECT 




-S110 



-S120 



-S130 



YES 





STORE IMAGE 
IN IMAGE BUFFER 







-S50 



S160 



SEND COMMAND TO 
DISPLAY, AT A DESIRED 
TIMING, IMAGES IN IMAGE 
BUFFER (DESIRED NO. OF 

FRAMES (n FRAMES 
BEFORE AND AFTER THE 
DESIRED TIMING)) 



S170- 



OUTPUT/DISPLAY 
BUFFERED IMAGES 



c 



RETURN 



30/ n 



27/32 




10/ "«9192 



28/32 



FIG.27(b) 



c 



START 



3 



ACQUIRE 
IMAGE 



PROCESS 
IMAGE 



-S210 



-S220 



COMPARE MATCHING 
RESULT WITH SET 
VALUE (1) 



-S230 




STORE IMAGE 



-S250 



SEND DISPLAY 
COMMAND 



DISPLAY/ 
OUTPUT 

( RETURN ^ 



-S260 



-S2T0 



10/089192 



29/32 



CD 



I •— • — I 

^cro 



oo 

OO 



U31U3ANOO "IVNDIS 







GO 






ZD 






CO 


H 




ND 






< 


o 






cr 






o 


M 


o 
o 


_i 

o 


W 


\ 


TR 




o 


z: 




p 


o 




o 


o 




=3 




cr 






h- 

00 









A-K 
W 



=5 0 0 0 



■ ujjnoaioi 



(0 

s o 
\ o 



j lziinomoj l 



CO 

o 



CM 




O 

00 
CO 
LU 

cr^ 
I 

LU 

o 
< 





en 






o 






b: 












o 









CD 



.CO 



CO 
CM 

O 



Ei nn rm -cu 
© E H 

a -a -a 

im rm LC3 



0 

_ 0 _ . 



■s -s: 
e a: 



o 




T — 

CN 



10/ 089-192 



30/32 



oo 



CD 



-Z.OL 
WZ.-Z. 

oo 

OQ 



I 



cL 

U31U3ANOO 1VNDIS 



o 
tr 

o 
_j 
o 
cr 

H 
O 

o 



A— \ 
W 



00 
ZD 
CO 

Q 
< 



o 




o 

V*" 

CM 



UJ _. 

a. 



A 0 0 0 



to 




-CO 



05 
CM 

a 
XL 



10/ 08919 



31/32 



N 



o 

CO 

d 



ZD 

o 
cr 

o 
_J 
o 



o 
o 



CD 



wzz 
oo 
oo 



cGlcGANOO 1VN0IS 



CD 



CD. 



T- 



A-K 

v-y 



00 
ZD 

m 

Q 

;z 
< 

o 
o 

-z. 
o 
p 
o 

ID 

h- 

00 



linodio 

0NISS300c: 


Id 


403 

(400) 




unociio 

0NISS300Ud 


Cvi CD 
CD O 


I 


unodio 

DNISS3QOcJd 


■< — s 

CD CD 



\CD Oy 
LO LC3 
CM OJ 



a 
■z 

CO 
CO 
LU 

Ol_ 
o~ 

I 

LU 
O 
< 



03 
r- 

d. , 

o 
t: 



CD_ 

LO 

OJ 



CM 

x 



LO 



CD 



O 
CM- 



O 
O 



32/32 



CD 



zo: 

oo 
oo 



aaiid3AN00 nvNDis 



T- 



/-A 
W 



unoaio 

DNISS300b 


\d 


403 
(400) 








unodio 
ONissaoodd 


402 

(400) 




1 


imodio 
ONissaooad 


5§ 
t 3 



o 

00 
CO 
UJ 

O h 

o~ 

i 

UJ 

O 
< 



r 



— QQ 




Docket No.: 

DECLARATION AND POWER OF ATTORNEY 
UBDER 35 USC § 35 (c)(4) FOR 
PCT APPLICATION FOR UNITED STATES PATENT 

As a below named inventor, I hereby declare that: 

my residence, post office address and citizenship are as stated below next to my name; 

I verily believe I am the original, first and sole inventor (if only one name is listed below) or an 
original, first and joint inventor (if plural names are listed below) of the subject matter which is claimed and for 
which a patent is sought, namely the invention entitled: CAMERA SYSTEM FOR HIGH-SPEED IMAGE 
PROCESSING described and claimed in international application number PCT/JPOO/06905 filed 
October 4, 2000 . 

I have reviewed and understand the contents of the above-identified specification, including the 
claims, as amended by any amendment referred to above. 

I acknowledge the duty to disclose to the Office all information known to me to be material to 
patentability as defined in Title 37, Code of Federal Regulations §1.56. 

Under Title 35, U.S. Code §119, the priority benefits of the following foreign application(s) filed 
within one year prior to my international application are hereby claimed: 

Japanese Patent Application No. 11-282969 filed October 4, 1999 
Japanese Patent Application No. 11-296753 filed October 19, 1999 

The following application(s) for patent or inventor's certificate on this invention were filed in 
countries foreign to the United States of America either (a) more than one year prior to my international 
application, or (b) before the filing date of the above-named foreign priority application(s): 

I hereby appoint the following attorneys of record with full power of substitution and revocation to 
prosecute this application and to transact all business in the Patent Office: 

James A. Oliff, Reg. No. 27.075: William P. Berridge, Reg. No. 30.024 : 
5 Kirk M. Hudson, Reg. No. 27,562; Thomas J. Pardini, Reg. No. 30,411; 

Edward P. Walker, Reg. No. 31^§ft; Robert A. Miller, Reg. No. 32^771; 
' y Mario A. Costantino, Reg. No. 33*565; and Caroline D. Dennison, Reg. No. 34.494 . 

ALL CORRESPONDENCE IN CONNECTION WITH THIS APPLICATION SHOULD BE SENT TO OTJFF 
& BERRIDGE , PLC, P.O. BOX 19928, ALEXANDRIA, VIRGINIA 22320 . TELEPHONE (703) 836-6400. 

I hereby declare that I have reviewed and understand the contents of this Declaration, and that all 
statements made herein of my own knowledge are true and that all statements made on information and belief 
are believed to be true; and further that these statements were made with the knowledge that willful false 
statements and the like so made are punishable by fine or imprisonment, or both, under Section 1001 of Title 18 
of the United States Code and that such willful false statements may jeopardize the validity of the application or 
any patent issued thereon. 

1 Typewritten Full Name 
I _ of Sole or First Inventor Haruvoshi TOYODA 

2 
3 





Given Name 


Middle Initial 


Family Name 


Inventor's Signature 








Date of Signature 


March ' 


20 ' 


2002 


Month 

Residence: Hamamatsu-shi r\ F»>c 


Day 


Year 
JAPAN 


Citizenship: JAPANESE 


City 


State or Province 


Country 



Post Office Address: do Hamamatsu Photonics K.K., 1 126-1, Ichino-cho, 

(Insert complete mailing 



address, including country) Hamamatsu-shi, Shizuoka 435-8558, JAPAN 



Note to Inventor: Please sign name on line 2 exactly as it appears in line 1 and insert the actual date of 
signing on line 3. 

IF THERE IS MORE THAN ONE INVENTOR USE PAGE 2 AND PLACE AN "X" HERE [X] 



(Discard this page in a sole inventor application) 



2 - 



3 - 



1 


Typewritten Full Name 
of Joint Inventor 


Kazuhiro 




NAKAMURA 






Given Name 


Middle Initial 


Family Name 


2 


Inventor's Signature 




Ai*kf mum 




3 


Date of Signature 


March 


20 


2002 



Residence: 
Citizenship: 



Month 

Hamamatsu-shi J f^Xl 



Day 



Year 
JAPAN 



City 



State or Province 



Country 



JAPANESE 



Post Office Address: 
(Insert complete mailing 
address, including country) 



c/o Hamamatsu Photonics K.K., 1126-1, Ichino-cho, 
Hamamatsu-shi, Shizuoka 435-8558, JAPAN 



1 


Typewritten Full Name 

of Joint Inventor Seiichiro 




MIZUNO 




Given Name," 


Middle Initial 


Family Name 


2 


Inventor's Signature ^JLl^Jwuv^ 






3 


Date of Signature Mar: ch 




2002 




Month 

Residence: Hamamatsu-shi .T f^OC 


Day 


Year 
JAPAN 




City 

Citizenship: JAPANESE 


State or Province 


Country 



Post Office Address: 
(Insert complete mailing 
address, including country) 



c/o Hamamatsu Photonics K.K., 1126-1, Ichino-cho, 
Hamamatsu-shi, Shizuoka 435-8558, JAPAN 



Typewritten Full Name 
of Joint Inventor 

Inventor's Signature 
Date of Signature 

Residence: 



Hiroo 



YAMAKAWA 



Given Name 


Middle Initial 


Family Name 


March 


20 


2002 



Month 

Hamamatsu-shi J~ f=>->c 



Day 



Year 
JAPAN 



City 



State or Province 



Country 



Citizenship: 



JAPANESE 



Post Office Address: 
(Insert complete mailing 
address, including country) 



c/o Hamamatsu Photonics K.K., 1126-1, Ichino-cho, 
Hamamatsu-shi, Shizuoka 435-8558, JAPAN 



Typewritten Full Name 
of Joint Inventor 



Inventor's Signature 
Date of Signature 

Residence: 



Given Name 



Middle Initial 



MUKOHZAKA 
Family Name 



March 



2002 



Month 

Hamamatsu-shi J p> X- 



Day 



Year 
JAPAN 



City 



State or Province 



Country 



Citizenship: 



JAPANESE 



Post Office Address: 
(Insert complete mailing 
address, including country) 



c/o Hamamatsu Photonics K.K., 1126-1, Ichino-cho, 
Hamamatsu-shi, Shizuoka 435-8558, JAPAN 



Note to Inventor: Please sign name on line 2 exactly as it appears in line 1 and insert the actual date of signing on line 3. 

This form may be executed only when attached to the first page of the Declaration and Power of Attorney of the application 
to which it pertains. 



